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ABSTRACT 
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provided services to rural areas or community segments most in need. Projects 
primarily served elementary and secondary students and adults. The majority of 
projects aimed at developing educational resources, providing training to 
students and other community residents, or installing telecommunications 
applications. The most common services offered were adult literacy, secondary 
education, basic skills, and math and science education. Almost three-quarters 
of survey respondents reported that their level of accomplishment met or 
exceeded their original expectations. Many projects were able to sustain 
themselves beyond their ARC grant, but not many had expanded significantly 
beyond their original mission. Recommendations were made to enable the ARC to 
further enhance its capacity to assist its educational projects. Performance 
goals should be clarified and expanded. The ARC should continue to enhance its 
application workbook and guidelines. Final report requirements should be 
standardized. The practice of conducting validation visits to a sample of ARC 
projects should be continued. Workshops in effective data collection and 
analysis strategies should be provided. The ARC database should be enhanced 
and its information dissemination practices reinforced. Seven extensive 
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EXECUTIVE SUMMARY 



This report summarizes findings from an evaluation of the implementation and impact of 84 
education projects funded by the Appalachian Regional Commission (ARC) during the 1990s. The 
study — conducted by Westat, a Rockville, Maryland, research firm — was designed to examine the range 
of activities that projects used to enhance learning opportunities, the extent to which these activities were 
implemented, the accomplishments associated with these activities, and the strategies that projects used to 
sustain themselves beyond the ARC grant period. Of particular interest was the extent to which projects 
achieved the outcomes set forth in their original proposals to ARC — and whether the activities and 
benefits associated with the ARC grants have been sustained over time. 



Overview of Educational Attainment in Appalachia and the ARC 

The Appalachian Regional Commission (ARC) was created in 1965 to promote economic 
and social development in the states and counties that made up the Appalachian Region. 1 The 
Commission represents a federal-state alliance designed to “advocate for and partner with the people of 
Appalachia to create opportunities for self-sustaining economic development and improved quality of 
life.” 2 In 1994, the ARC began a broad-based strategic planning process, ultimately yielding a strategic 
plan for 1997-2002. The strategic plan clarifies the Commission’s role in promoting progress in the 406 
counties of the Appalachian Region and highlights five main goals: 

■ Appalachian residents will have the skills and knowledge necessary to compete in the 
world economy of the 21 st century. 

■ Appalachian communities will have the physical infrastructure necessary for self- 
sustaining economic development and improved quality of life. 

■ The people and organizations of Appalachia will have the vision and capacity to 
mobilize and work together for sustained economic progress and improvement of their 
communities. 



1 The Appalachian Regional Development Act, as amended in 1998, identifies the 406 counties of the Appalachian Region. These counties 
include all of West Virginia and parts of Alabama, Georgia, Kentucky, Maryland, Mississippi, New York, North Carolina, Ohio, Pennsylvania, 
South Carolina, Tennessee, and Virginia. 

2 Setting a Regional Agenda, ARC Strategic Plan: 1997-2002. 
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Appalachian residents will have access to financial and technical resources to help 
build dynamic and self-sustaining local economies. 

Appalachian residents will have access to affordable, quality health care. 



The first strategic goal is the focus of this study. There are two specific objectives for this 
goal (with the second being the focus of this study): 

■ The percentage of workers receiving basic education and skills training, skills 
upgrading, and customized training will increase, leading to development of a 
workforce that is competitive in the 21 st century world economy. 

■ The percentage of students participating in school readiness, dropout prevention, 
school-to-work transition, and GED programs will increase, thereby raising the 
college-going rate and preparing students for the world of work in the 21 st century. 



Study Overview 



In summer 1999, Westat initiated an evaluation of a sample of educational projects that 
received ARC funding during the 1990s. 3 A primary purpose was to assess the extent to which projects 
were able to achieve the outcomes they delineated in their original proposals to ARC. Also of interest was 
whether project-level gains had been sustained beyond the period of the ARC grant. The following 
evaluation questions guided the overall conduct of the study: 

■ What are the characteristics of communities and individuals that benefited from the 
projects? 

■ What problems were projects designed to address? 

■ What approaches did the projects use to ameliorate these problems? 

■ What specific outcomes were the projects designed to achieve? 

■ To what extent have projects accomplished their objectives? 

■ To what extent were project-related gains sustained beyond the period covered by the 
ARC grant? 

■ What factors influenced the projects’ ability to implement their approaches and 
achieve their objectives? 



3 This report only examines a subset of the types of education projects funded by ARC. Vocational education and workforce training projects will 
be examined in a subsequent report. 
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The evaluation utilized the following data collection approaches to derive quantitative and 
qualitative assessments of the impact of ARC’S investment in educational projects: 

■ A review of existing project and state documents and materials to inform the 
development of survey instruments and site visit interview protocols. The document 
review was also used to identify projects’ anticipated goals and objectives. 

■ Interviews with ARC staff, including project coordinators and the program operations 
director, to gain a better understanding of the application process and project 
management. 

■ Telephone interviews with state-level and Local Development District (LDD) level 
ARC coordinators. The interviews were designed to enhance our understanding of the 
educational priorities of the 13 states that compose the Appalachian Region. 

■ A mail survey to 91 ARC education projects that received funding during the 1990s. 
(The 84 projects that responded to this mail survey eventually constituted the actual 
study sample.) This survey was designed to collect a common set of data regarding 
projects’ characteristics, implementation practices, outcomes, and sustainability. A 
customized addendum to the survey collected narrative information on the extent to 
which a project’s original outcomes were achieved. 

■ Site visits to eight projects that had successfully provided community residents with a 
new or enhanced educational service. The purpose was to obtain more detailed 
information about unique and sustainable practices that might be replicated elsewhere. 



Characteristics of the Study Sample 

According to its Strategic Plan, part of ARC’S mandate is to “address economic 
underdevelopment that is deeply rooted in regionwide, historic patterns of isolation, exploitation, and 
outmigration.” Study findings suggest that the projects in the study sample contributed to the 
Commission’s goal of reaching those segments of Appalachia that are geographically isolated. Most 
projects provided services to rural areas — with almost half (48 percent) being situated in a region that was 
entirely nonmetropolitan. Equally significant, the community segments that were targeted generally 
reflected ARC’S goal (as articulated in both its strategic plan and the provisions of its Distressed County 
Program) of serving those most in need — including persons in extreme poverty (82 percent), persons who 
are geographically isolated (77 percent), those who are unemployed or underemployed (70 percent), 
youth at risk of dropping out of school (60 percent), and persons with disabilities (59 percent). In 
addition, one-third (3 1 percent) of projects were serving areas that included distressed counties. Although 



half of the projects in the study sample did not serve a distressed county, 4 their efforts were still focused 
on those community segments most in need of services. Many of the residents in these nondistressed 
areas faced similar educational and economic barriers as residents in distressed areas, and therefore 
required similar assistance from ARC. 

Projects in the study sample primarily served elementary and secondary students (48 
percent) and adults (33 percent). A small percentage of projects serve preschool students (7 percent) and 
communities as a whole (12 percent). Among projects serving elementary and secondary students, the 
vast majority were addressing the needs of all students (38 percent of all projects), rather than of a 
targeted group of students (10 percent of all projects). 

A comparison of projects in the study sample to the universe of all ARC projects funded 
during the same period found that the sample and the universe were roughly similar in economic and 
metropolitan status, ARC grant amount, and total project cost. All ARC states except Maryland (only 2 
percent of all education projects) were included in the study sample. Georgia and Kentucky were slightly 
overrepresented in the sample, while New York was slightly underrepresented. The mix of ARC project 
types in the sample varied only slightly from those of all education projects. Adult literacy, basic skills, 
and math/science education were overrepresented in the study sample, while distance learning was 
underrepresented in the sample. Child care and telecommunications were absent from the sample 
altogether. Finally, we were unable to match project types used for the study with those of all education 
projects due to the reclassification of the sample into adult, elementary/secondary (all students), 
elementary/secondary (targeted students), preschool, and communitywide. 



Project Activities 

The projects in the study sample reported conducting a total of 682 distinct activities — 556 
of which were partially or fully funded by ARC. The majority of projects conducted an ARC-funded 
activity aimed at developing educational resources (78 percent of projects), providing training to students 
and other community residents (71 percent), and/or installing/enhancing telecommunications applications 
(64 percent). The average educational initiative conducted activities in three different categories — 
suggesting that projects were fairly ambitious in the diversity of activities they chose to implement. 



4 



The remainder of projects were statewide or multistate and were not assigned an economic designation. 
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Across all activity types, the most frequently cited ARC-funded undertakings were installing computers 
(5 1 percent of projects), providing academic skills training (46 percent), providing computer training (45 
percent), providing teacher or tutor training (45 percent), and providing literacy training (37 percent). 

Over four-fifths (85 percent) of respondents met or exceeded their implementation 
objectives. Only 15 percent indicated that their level of implementation on at least one task was less than 
planned — and no projects reported that they implemented all of their activities less than planned. Nearly 
all of the 556 ARC-funded activities were implemented as planned — in fact, 21 percent were 
implemented more than planned. Finally, the surveys and case studies provide little evidence that projects 
encountered any widespread barriers that affected the degree to which an activity was implemented. With 
the exception of underestimating the time/effort needed to conduct the project, no more than one-fifth of 
respondents cited any single barrier as having hindered their capacity to carry out the activities in their 
ARC grants. 



Project Outcomes 

Projects were successful in achieving the outcomes they set forth in their original requests 
for ARC support. Almost three-quarters of survey respondents reported that their level of accomplishment 
met or exceeded their original expectations. Just under half (47 percent) of the 203 outcomes that projects 
identified in their ARC proposals were ultimately achieved the same as planned, while 30 percent were 
achieved more than planned and 13 percent were achieved less than planned. Skeptics might discount 
some of these claims of success as being based on self-report. However, documentation from the mail 
surveys suggests that many of these claims were, in fact, based on either a study (e.g., pre- and post- 
intervention assessment of participants’ basic skill levels) or external data source (e.g., students’ scores on 
tests, school dropout rates, unemployment rates). Further, the case studies uncovered convincing evidence 
that the efforts of the projects in the study sample resulted in a broad range of educational, economic, and 
social gains. 



Another important finding from our analysis of project outcomes was that over four-fifths 
(86 percent) of the projects in the study sample were, in fact, able to describe at least one anticipated 
outcome or community benefit in their proposal to ARC. (As stated above, the document review was used 
to extract and categorize the outcomes projects initially set out to achieve.) These outcomes, which we 
categorized into three general categories (educational, economic, and family/individual well-being), 
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outlined a skill or circumstance that would be enhanced as a result of their ARC grant. However, only a 
few of the outcomes described in the application materials had a quantifiable benchmark against which 
project success could be measured. Most projects used more general — and less quantifiable — language to 
describe their desired community goals. In addition, a small number of projects outlined a lofty goal that 
was beyond the possibility of a single project to achieve. Nonetheless, the fact that most projects had 
identified an outcome — and half appeared to have access to tangible evidence to back up their claims of 
success — suggests that a framework exists for ARC to build upon as it seeks to enhance its program 
monitoring capacities. 

Finally, survey respondents were asked to hypothesize what would have happened if their 
project had not received federal funding through ARC. Two-thirds (67 percent) indicated that their 
projects would never have been implemented without their ARC award. The remaining 27 respondents 
indicated that their projects probably would have been implemented using alternative funding sources. 
However, the majority of these 27 projects indicated that without ARC funding their projects would have 
offered dramatically fewer services (67 percent), reached significantly fewer people (56 percent), and 
suffered substantial delays in implementation (52 percent). 



Project Sustainability and Expansion 

Findings from both the survey and case studies suggest that many of the projects in the study 
sample were able to sustain themselves beyond their ARC grant. At the time the mail survey was 
administered, two-thirds of projects were still in full operation and/or had grown or expanded — and 
almost all projects reported that at least some of their ARC-fimded activities or equipment were still in 
use. 



While projects appeared to have sustained themselves, it did not appear that many had 
expanded significantly beyond their original mission. Thirty projects were serving more beneficiaries than 
originally intended, although the characteristics of these beneficiaries had basically remained the same. 
Additionally, 38 projects had generated spinoff activities, but most of these actually constituted an 
expansion in the number of persons served. 

Several factors appeared to be responsible for the high proportion of projects that remained 
operational or had expanded over time. In most cases, the general demand and need for the services were 
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still present or had grown — i.e., projects had expanded because there were simply more people who 
needed their services. Many projects also reported that they had enough financial and community support 
to allow them to continue or expand. Quality services, dedicated staff, strong leadership, and 
technological advances were also considered facilitators of expansion. While most of these projects were 
sustained, in many cases that result seemed serendipitous and was achieved without a high degree of 
planning. This finding suggests that project coordinators might encourage prospective projects to think 
ahead about steps they will take to sustain their efforts beyond the ARC grant period. 



Recommendations 



The lack of any prominent implementation barriers suggests that grant recipients were 
generally satisfied with the level of the financial and technical support they received from ARC. 
Similarly, case study participants praised the role that ARC played in making their projects possible. The 
mail survey, case studies, and telephone interviews with state and LDD staff were not explicitly designed 
to assess whether ARC staff were successful in promoting project-level success. Nonetheless, the absence 
of criticism among some of the Commission’s key stakeholders represents an important and encouraging 
study finding. 



The ultimate aim is for ARC to identify exemplary models of education improvement — and 
to develop strategies for duplicating these models throughout the Appalachian Region. In order for this to 
occur, ARC needs timely access to information about what is happening in the field. It also needs valid 
and reliable data on the types of educational interventions that are most likely to produce tangible results. 
The Commission now requires that its grant recipients provide detailed information about their 
anticipated and actual outcomes. (These requirements predate most of the projects in the study sample.) 
The following represent steps ARC might take to further enhance its capacity to monitor and assist its 
educational projects and to document the extent to which projects are meeting their own objectives and 
furthering the Commission’s strategic goals: 

■ Clarify and expand upon the performance goals for Goal 1. The lone performance 
goal for Goal 1 focuses on a broad output (“Provide access to education and training 
services to 45,000 students/trainees as a step to ensuring an educated, resourceful, and 
skilled population”), as opposed to a benefit that could be expected to occur if an 
education intervention is successful. We therefore recommend that ARC develop 
additional performance goals that might be incorporated into future versions of the 
ARC Strategic Plan. These additional goals could build upon the educational, 
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economic, and family/individual well-being outcomes that were reported on the mail 
survey. 

Continue to enhance the ARC’S application guidelines. In 1998, the Commission 
published an application workbook and application guidelines designed to improve 
the quality and consistency of the proposals submitted to ARC. 5 While this generic 
framework is sufficient, we would recommend that the Commission provide 
additional guidance and more explicit examples as to what constitutes an acceptable 
request for ARC assistance. The purpose would be to compel applicants (and ARC 
project coordinators) to think ahead about measuring project success and maximizing 
the likelihood of long-term sustainability. A secondary purpose would be to devise a 
framework for collecting standardized information from all applicants that could be 
entered into ARC’s database. 

Standardize ARC’S final report requirements. The most recent ARC Grant 
Administration Manual contains general guidelines and an example of topics that 
projects might address in their final report. However, few of the projects in the study 
sample provided the level of detail suggested by this model. We therefore recommend 
that ARC require that all projects use these guidelines when completing their closeout 
report. We also recommend that ARC provide customized examples of outputs and 
outcomes for each of the Commission’s five strategic goals. Finally, projects should 
be encouraged to describe the methodologies that were used to document project 
success in the project outcome section. 

Continue conducting validation visits to a sample of ARC projects. The 
Commission recently began conducting validation visits to projects that are no longer 
receiving ARC support. The purpose of these visits is to assess whether projects 
ultimately attained their longer term outcomes. In addition to helping uncover tangible 
evidence of project success, these visits have provided project coordinators additional 
justification for requiring that applicants identify outcomes that can be expected to 
occur several years after the end of their ARC grant period. We believe that these 
visits serve an important function — especially if they compel projects to consider how 
their efforts will be assessed in future years — and should therefore be continued. 

Provide workshops in effective data collection and analysis strategies. Findings 
from both the survey and case studies suggest that projects had difficulty 
conceptualizing and operationalizing even the most basic data collection activities. 
This lack of evaluation data ultimately hinders ARC’s ability to assess the 
effectiveness and impact of its education projects. Equally important, in an age of 
educational accountability, a project that lacks credible evaluation data may be at a 
disadvantage as its seeks funding from other sources. We recommend that ARC 
consider holding evaluation workshops with its LDDs and other interested 
stakeholders on a periodic basis — with particular emphasis on techniques for 



5 Most of the projects in the study sample were funded before ARC had revised its application guidelines or published the ARC Project 
Application Workbook. However, our preliminary review of grant proposals submitted under these new guidelines (for an upcoming study of 
ARC’s vocational education and workforce training projects) suggests that they have had the intended effect of increasing the amount of 
pertinent information provided in applications and improving the quality of anticipated outcomes. 
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collecting baseline and follow-up data on project participants’ educational and 
economic status. 

Enhance the ARC database. The ARC database contains a wealth of information 
about the projects that have received Commission funding over time. However, our 
experience in using this resource suggests that some of the data that ARC collects 
from its grant recipients are either incomplete or inaccessible. This has resulted in an 
underutilization of an important resource tool for systematically monitoring projects’ 
activities and outcomes. We therefore recommend that ARC (1) add a field in the 
ARC database that links Commission-funded projects with successor state-funded 
projects, (2) enter information on the economic designation and metropolitan status of 
all counties that a project is serving, and (3) divide project type into two separate 
codes — one for the population that is to be served, and one for the overall approach or 
type of service that is to be provided. 

Reinforce ARC’S dissemination practices. The Commission’s Strategic Plan 
specifies that ARC’S national staff are to serve as information brokers within the 
Region by disseminating information to stakeholders through studies, public forums, 
and workshops. ARC is already using several mechanisms to disseminate information 
about successful projects, including articles about specific projects in Appalachia 
Magazine , annual “best practices” conferences, and technical assistance with 
individual projects. While these approaches are all valuable, there are additional steps 
that ARC might take to enhance the level of information (e.g., effective 
implementation strategies, cost-effective approaches for collecting and analyzing 
outcome data) that can be made available to current and future grant recipients. One 
approach would be to develop an on-line clearinghouse for LDDs and communities 
looking for ideas about how best to address an educational barrier. 



1. INTRODUCTION 



Although educational attainment in Appalachia has increased in the last 30 years, stubborn 
problems associated with illiteracy, lack of access to resources (both educational and technological), lack 
of appropriate child development programs, and lack of services for at-risk youth persist. While the 
region is growing stronger overall, there are pockets where considerable progress is still needed. 
According to the Request for Proposal: 

Considerable educational deficits... remain in portions of the Region, 
particularly throughout much of central Appalachia, where the average high 
school completion rate for youth 18 to 24 years old is only 68 percent. Adult 
high school completion rates are at the national average for metropolitan 
counties of northern Appalachia, but are still only 60 percent for rural southern 
Appalachia and only 52 percent for rural central Appalachia. 



Moreover, as stated in the Appalachian Regional Commission’s strategic plan: 1 



More than 4 million adult Appalachians do not have a high school diploma or GED; 2 
million have less than a ninth-grade education. These Appalachians are not equipped 
to take advantage of technical or skills training. 

Many areas of the Region — particularly in central Appalachia — have dropout rates of 
more than 40 percent. Regionwide, 6 1 counties have rates that exceed 50 percent. The 
ripple effect of high dropout rates is a greater demand on social services and an 
increase in the number of working poor. 

Most of the 50 percent of Appalachian students that are not college bound do not have 
the critical thinking and computer skills required by today’s techno logy -oriented 
employers. 

Rural counties in Appalachia are twice as likely to be considered “severely distressed” 
as rural counties elsewhere in the nation. 

The geographic and cultural isolation that led the Region to fall behind in the nation in 
economic development is also an impediment to its participation in the globalization 
of the nation’s economy. 



This report summarizes findings from an evaluation of the implementation and impact of 84 
education projects funded by the Appalachian Regional Commission (ARC) during the 1990s. The 
study — conducted by Westat, a Rockville, Maryland, research firm — was designed to examine the range 

1 Setting a Regional Agenda, ARC Strategic Plan: 1997-2002. 
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of activities that projects used to enhance learning opportunities, the extent to which these activities were 
implemented, the accomplishments associated with these activities, and the strategies that projects used to 
sustain themselves beyond the ARC grant period. Of particular interest was the extent to which projects 
achieved the outcomes set forth in their original proposals to ARC — and whether the activities and 
benefits associated with ARC grants have been sustained over time. 

This chapter provides an overview of ARC, the steps it has taken in recent years to enhance 
educational attainment throughout the Region, and the process by which the Commission determines 
which initiatives should receive grant funding. It also provides an overview of the purpose and conduct of 
the evaluation study. 



1.1 Overview of the Appalachian Regional Commission 



The Appalachian Regional Commission was created in 1965 to promote economic and social 
development in the states and counties that compose the Appalachian Region. 2 The Commission 
represents a federal-state alliance designed to “advocate for and partner with the people of Appalachia to 
create opportunities for self-sustaining economic development and improved quality of life.” 3 The 
strategic plan clarifies the Commission’s role in promoting progress in the 406 counties of the Region: 



ARC’s role is to work collaboratively in effectively targeting resources to 
achieve the vision. The Commission’s intention is to equip the people and 
communities of Appalachia with the entrepreneurial skills and enterprise 
development resources they need to create self-sustaining local 
economies. ..ARC’s mandate is to address economic underdevelopment that is 
deeply rooted in regionwide, historic patterns of isolation, exploitation, and 
outmigration. ARC is specifically charged with creating regional solutions to 
pervasive economic problems that reach broadly across state lines. 



The Appalachia Regional Development Act (ARDA) established the Appalachian Regional 
Commission as a partnership between the governors of the 13 states in the Region and a presidential 
appointee representing the federal government. The federal appointee and one of the governors act as 
co-chairs of the Commission. Within each state, local development districts (LDDs) provide for 



2 The Appalachian Regional Development Act, as amended in 1998, identifies the 406 counties of the Appalachian Region. These counties 
include all of West Virginia and parts of Alabama, Georgia, Kentucky, Maryland, Mississippi, New York, North Carolina, Ohio, Pennsylvania, 
South Carolina, Tennessee, and Virginia. 

3 Setting a Regional Agenda, ARC Strategic Plan: 1997-2002. 
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grassroots-level participation, so that ARC activities originate from and ultimately benefit the 
communities. The ARDA focuses on ameliorating the widespread poverty in Appalachia by making 
investments in the education and job skills of community residents. The Act specifically directs ARC to: 

■ Promote federal-state cooperation in devising a regional approach to development; 

■ Provide the major portion of funding for a regional highway system to alleviate the 
Region’s isolation; 

■ Work with existing federal and state agencies to build the broad array of community 
and human development programs needed to correct the enormous shortfalls in 
education, health care, and other services that were the legacy of decades of neglect; 

■ Help develop the local leadership that is essential to the success of the federal-state 
partnership at the community level; and 

■ Serve as an advocate and broker for the Region with public and private organizations 
to ensure coordination of all available resources to support Appalachia’s development. 



1.2 ARC Strategic Goals 



In 1994, ARC began a broad-based strategic planning process, ultimately yielding a strategic 
plan for 1997-2002. This plan includes five strategic goals directed at improving the local and regional 
economy through education, infrastructure (physical and organizational), and access to resources and 
health care (Exhibit 1-1). The first strategic goal of ARC — and the focus of this study — is that the people 
of Appalachia will develop the “skills and knowledge necessary to compete in the world economy of the 
2 1 st century.” There are two specific objectives for this goal: 4 



The percentage of workers receiving basic education and skills training, skills 
upgrading, and customized training will increase, leading to development of a 
workforce that is competitive in the 21 st century world economy. 

The percentage of students participating in school readiness, dropout prevention, 
school-to-work transition, and GED programs will increase, thereby raising the 
college-going rate and preparing students for the world of work in the 2 1 st century. 



4 This evaluation did not examine the vocational education or workforce training projects that fall under these objectives. 
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Exhibit 1-1. ARC strategic goals, 1997-2002 



1 . Appalachian residents will have the skills and knowledge necessary to compete in the world economy 
of the 21 st century. 

2. Appalachian communities will have the physical infrastructure necessary for self-sustaining economic 
development and improved quality of life. 

3. The people and organizations of Appalachia will have the vision and capacity to mobilize and work 
together for sustained economic progress and improvement of their communities. 

4. Appalachian residents will have access to financial and technical resources to help build dynamic and 
self-sustaining local economies. 

5. Appalachian residents will have access to affordable, quality health care. 



In the years since most of the study sample projects were funded, ARC has taken steps to 
further define these objectives in an application workbook provided to State Program Managers. New 
guidelines delineate output and outcome measurements for each objective. Whenever possible, ARC is 
requiring the use of these measurements in its grant applications. In addition, project coordinators provide 
periodic training to LDDs in the use of this workbook. 

For the first objective (workers), the output measure is the number of workers who 
participate in a skill activity, and outcome measures are the number of workers who obtain employment 
or retain employment. ARC’S progress with respect to these outputs and outcomes is examined in a 
subsequent evaluation report that focuses on the Commission’s vocational education and workforce 
training projects. 

The second objective (students) pertains to the projects in the study sample. The output 
measure is the number of students participating in the educational activity. Outcome measures include the 
number of students who: 

■ Obtained a GED within 1 year; 

■ Graduated from high school (at-risk dropouts); 

■ Completed a school readiness activity and tested on grade level entering kindergarten; 



■ Entered the workforce after participating in a school-to-work program within 1 year 
(high school/college students). 

The Strategic Plan delineates several broad strategies for developing the skills and 
knowledge needed to compete in the current economy. These approaches include (1) developing 
partnerships with other agencies, including the U.S. Department of Energy (e.g.. Oak Ridge National 
Laboratory), the National Science Foundation (e.g., Appalachian Rural Systemic Initiative), and the 
Southern Regional Education Board (e.g., “High Schools that Work” program), (2) forging partnerships 
with states aimed at identifying areas of need and developing targeted interventions, (3) developing 
private-public national and regional collaborations, (4) using advanced telecommunications to enhance 
teacher training and worker preparation initiatives, and (5) identifying exemplary models of education 
improvement and strategies to duplicate them throughout the region. In addition, the Plan identifies one 
performance measure for Goal 1: “Provide access to education and training services to 45,000 
students/trainees as a step to ensuring an educated, resourceful, and skilled population.” 



1.3 Process by Which ARC Grants Are Awarded 

To accomplish its five strategic goals, ARC provides financial and technical support to local, 
regional, and multistate projects through its Area Development Programs. The process for awarding these 
grants reflects the underlying partnership between the Commission and participating states, as well as the 
need to give local communities a voice in determining how ARC funds are to be allocated. As such, most 
of the projects included in the study sample originated at the local or state level. Many were forwarded to 
the attention of ARC after a local or statewide competitive process. 

Each year, the 13 states of Appalachia prepare individual annual strategy statements and 
spending plans. These documents contain state-level goals (which are aligned with ARC’S five strategic 
goals) and corresponding proposals for each of the specific initiatives that are being recommended for 
funding. In some states, these initiatives are developed — in consultation with the LDDs — to reflect state 
and/or local priorities. In others, applicants submit proposals to their LDD or directly to the state and 
subsequently participate in a competitive review process. Our interviews with LDD and state-level staff 
revealed that no two states use the same method for determining which projects should be proposed for 
funding, and each LDD works under varying degrees of autonomy and provides vastly different amounts 
of technical assistance both before and after an ARC grant is awarded. 
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Once approved by the governor, a state’s recommendations for project funding are submitted 
to ARC. Each proposed project is then reviewed by ARC project coordinators and, in most cases, 
approved by the federal co-chair. Project coordinators can negotiate changes to the proposed project with 
State Program Managers. In most cases, adjustments are made to timetables and budgets. More recently, 
negotiations have surrounded the substance and specificity of outputs and outcomes. A limited number of 
projects originate and are funded each year directly through the Commission and ARC set-asides. These 
projects are subject to the same policies and procedures as those funded through individual states. 



1.4 Program Changes 

Over the past several years, the ARC has made some changes to its application, proposal 
review, and program monitoring processes. First, as described previously, program staff developed a 
workbook for state program managers and applicants with the intent of collecting more complete 
application packages. 5 By providing examples of outputs and outcomes, they hoped to encourage 
prospective projects to be mindful of these concepts when designing their implementation plans and to 
identify specific outputs and outcomes in their grant proposals. Second, staff provided a Grant 
Administration Manual that describes what should be included in quarterly progress reports and a 
project’s final report. The manual includes sample formats and examples of how output and outcome 
measures can fit into the narratives. Program staff are also taking a greater role in negotiating with states 
and projects to improve the quality of the projects by requiring that outcomes be more specific. Most 
recently, ARC staff have begun making site visits to a sample of projects 2 years after the end of their 
grant period. These validation visits are designed to assess whether projects eventually achieve their 
longer-term outcomes. 6 

Early work for a subsequent study on ARC’s vocational education and workforce training 
projects has shown that these changes have, in fact, made a difference. A comparison of documentation 
from projects funded in the early 1990s to projects funded more recently shows a much greater emphasis 
on outcomes in grant applications. Because findings and recommendations drawn from this evaluation are 



5 The ARC Project Application Workbook contains a number of useful items to help state staff develop their own application packages and 
ensure complete documentation, including grantee checklists, internal checklists, and required federal forms. 

6 As is discussed in Study Methodology (Section 1.6), these requirements were put in place after most of the projects in the study sample had 
been awarded. 




1-6 



27 



reflective of a program that has changed, we do not attempt to contextualize these findings to the current 
system. 



1.5 Study Overview 

In summer 1999, Westat initiated an evaluation of a sample of educational projects that 
received ARC funding during the 1990s. A primary purpose was to assess the extent to which projects 
were able to achieve the outcomes they delineated in their original proposals to ARC. Also of interest was 
whether projects’ gains had been sustained beyond the period of the ARC grant. The following evaluation 
questions guided the overall conduct of the study: 

■ What are the characteristics of communities and individuals that benefited from the 
projects? 

■ What problems were projects designed to address? 

■ What approaches did the projects use to ameliorate these problems? 

■ What specific outcomes were the projects designed to achieve? 

■ To what extent have projects accomplished their objectives? 

■ To what extent were project-related gains sustained beyond the period covered by the 
ARC grant? 

■ What factors influenced projects’ ability to implement their approaches and achieve 
their objectives? 

The evaluation utilized a number of data collection approaches to derive quantitative and 
qualitative assessments of the impact of ARC’S investment in educational projects. These approaches 
were applied to a sample of the projects that began receiving ARC funding between 1 990 and 1 998. 7 The 
following data collection strategies were used to address these evaluation questions: 

■ A review of existing project and state documents and materials to inform the 
development of survey instruments and site visit interview protocols. Findings from 
the document review were also used to generate a customized addendum to the survey 
that contained each project’s anticipated outcomes. (Respondents were then asked to 
provide information about the extent of achievement for each anticipated outcome 



7 Projects initiated between 1990 and 1998 were eligible for inclusion in the evaluation because, at the time the mail survey was administered 
(January 2000), enough time had passed to assess these projects* accomplishments and longer term sustainability. 



identified in the document review.) Finally, findings from the document review were 
used to augment our analysis of survey data. 

■ Interviews with ARC staff, including project coordinators and the program operations 
director, to gain a better understanding of the application and management procedures 
at ARC and within individual states. 

■ Telephone interviews with state-level and LDD-level ARC coordinators. The 
interviews were designed to enhance our understanding of the educational priorities of 
the 13 states of Appalachia. Other topics covered during these telephone interviews 
included the state-level process by which ARC grants are awarded and the monitoring 
practices that states use to determine whether a given project achieved its objectives. 
Finally, the interviews were used to obtain recommendations for potential case study 
sites and to identify additional study issues of interest to state and local staff. 

■ A mail survey to 91 ARC education projects that received funding during the 1990s. 
(The 84 projects that responded to this mail survey constitute the actual study sample.) 
This survey was designed to collect a common set of data regarding projects’ 
characteristics, implementation practices, outcomes, and sustainability. It also 
obtained extensive narrative information on the extent to which a project’s original 
outcomes were achieved. 

■ Site visits to eight projects that had successfully provided community residents with a 
new or enhanced educational service. The purpose was to obtain more detailed 
information about unique and sustainable practices that might be replicated elsewhere. 
Site visit findings were summarized in case studies and, where appropriate, used in the 
final report. 

Appendix D provides a more detailed overview of these activities, as well as a discussion of 
the procedures used to select and refine the study sample. Appendix E provides information on the 
process used to select the eight case study sites. 



1.6 Study Methodology 

This report summarizes findings from the various study activities, with particular emphasis 
on the mail survey and document review. Several caveats regarding the study are worth noting. First, as is 
discussed in Appendix D, the process used to select the study sample systematically excluded initiatives 
that lacked a complete project file at ARC headquarters (in some cases, project files were in the closure 
process or under internal review and were not available for the evaluation) and/or a knowledgeable 
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individual who could respond to the mail survey. 8 These exclusions, while necessary, increased the 
likelihood that we would primarily survey projects that successfully implemented their ARC grant — and 
potentially limited our opportunity to examine factors that can hamper the efforts of ineffectual projects. 
In addition, projects that received less than $10,000 were excluded from the sample. Any findings 
regarding the success and sustainability of ARC-funded educational projects are therefore limited to the 
84 initiatives that responded to the mail survey. 

Second, it is worth emphasizing that any references in this report to site visit findings reflect 
a purposefully selected segment of the study sample — i.e., those projects that appeared to have been able 
to sustain an innovative or unique practice over time. As such, any conclusions drawn from the site visits 
may not pertain to the overall study sample. 

Third, the RFP called for an analysis by project type. However, the project types assigned by 
ARC pertain both to the population being served and to the type of services being provided: adult literacy, 
basic skills, distance learning, dropout prevention, elementary education, preschool, educational 
partnerships, math/science education, and secondary education. As such, these designations were not 
particularly useful for the purpose of comparing projects. In an effort to develop alternative designations 
that would allow for more meaningful comparisons, we used the document review to assign projects to 
one of five categories designed to reflect the population receiving services — i.e., preschoolers, all 
elementary/secondary students, targeted elementary/secondary students, adults, and communitywide. 

Fourth, as discussed in Section 1.4, ARC has made some recent improvements to its 
application and reporting guidelines to enhance the collection of outcome information. A brief review of 
FY 2000 project documentation for a subsequent report on vocational education and workforce training 
projects indicates that the more recent projects have benefited from these changes. Nevertheless, the 
changes were not in effect for the study sample and, therefore, are not discussed with the study findings. 

Finally, the RFP called for an assessment of impacts after controlling for differences among 
project characteristics (e.g., metropolitan designation or economic status of participating communities). 
However, given the small number of projects that responded to the mail survey, we were somewhat 



K The decision to exclude initiatives that appeared unlikely or unable to respond to the mail survey reflected a concern that we would not obtain 
sufficient information from a significant proportion of respondents if the study sample comprised randomly selected projects. This concern was 
exacerbated by three factors — i.e., the considerable length of time that had elapsed since some of the projects had received ARC funding, the 
lack of sufficient documentation for some projects at ARC’s headquarters, and the small number of projects (91) included in the original study 
sample. 
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limited in our ability to disaggregate specific findings (e.g., degree to which specific objectives were 
achieved) across multiple project characteristics. The problems associated with the small sample size 
were exacerbated by the low cell counts that occurred when the survey responses were divided according 
to a variety of project characteristics. Typical statistical standards require at least five cases in each cell of 
a multivariate procedure, although ten or more cases in each cell results in more powerful analyses. For 
example, six projects in the study sample were identified as preschool projects. Conducting any analyses 
of these six projects resulted in cells sizes of less than five. Additionally, because many of these project 
characteristics were themselves intercorrelated, the data were not appropriate for high-level, complex 
regression analyses. 

This issue was handled in two ways. First, although we were hesitant to rely extensively on 
multivariate analyses given the small sample size, univariate analyses still yielded important information 
about differences between projects on single, oftentimes critical, dimensions. Thus, in the text of this 
report, we provide information about projects based upon several descriptive characteristics (see below), 
and conduct appropriate univariate or bivariate analyses to test for differences. 9 All comparative 
statements based upon these tests are significant at the 0.10 level or better. While use of a critical p value 
of 0.10 represents a more liberal criterion than the generally employed level of 0.05, we believe that given 
the study’s small sample size (84), such an adjustment allows for the identification of statistically 
significant differences that are large, but would otherwise be missed. Throughout the report, only 
noteworthy differences that are statistically significant are called out in the text. In addition, because 
comparisons within tables were often made both vertically and horizontally, indications of whether 
differences across rows or columns were statistically significant would have been extremely dense and 
potentially misleading. We have therefore not indicated any statistical differences in the tables. 

Second, Appendix C contains tables for a series of additional analyses that are not discussed 
extensively in body of the report because small, and often uneven, cell sizes greatly limit the power of the 
analyses. However, illustrative findings from these analyses are reported in the main text, especially 
where trends are apparent. These tables may be helpful to stakeholders interested in examining key 
findings by one of the following project characteristics: 

■ Project type — i.e., preschool age, elementary/secondary (all students), 

elementary/secondary (targeted students), adults, and communitywide. The indication 
of project type is important for individuals with specific interests in different levels of 



9 All comparative statements were tested using appropriate statistical analyses, typically t-tests, ANOVAs, or Chi square with appropriate post 
hoc tests. 
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intervention (e.g., school or communitywide) or different targets of intervention (e.g., 
all students or selected students). This characteristic identifies who was targeted by 
project activities with some indication of the context for those activities. 

Project scope — i.e., project served a single town or county, project served two or 
more adjacent counties, and project served two or more nonadjacent counties. This 
characteristic provides an indication of whether a project represented a localized effort 
(e.g., a single school) or was intended to benefit individuals in a far larger community 
(e.g., statewide or extending beyond state boundaries). This characteristic may be of 
special interest to policymakers in judging the potential reach of a project, as well as 
individuals proposing to implement activities. For example, data on project scope 
might provide some insight into what types of activities and approaches “work” on 
different scales. 

Economic status — i.e., project served at least one distressed county, project served no 
distressed counties, and project served all ARC counties within a state (or served 
multiple counties in two or more states). All counties in the Commission’s domain are 
assigned an economic classification. Counties identified as “distressed” show the most 
dire economic circumstances. This characteristic is therefore of direct policy relevance 
to ARC, but also to potential grantees who may question how a project’s impact is 
influenced by the local economy. 

Metropolitan status— i.e., project only situated in nonmetropolitan counties, project 
only situated in metropolitan counties, project situated in a combination of 
nonmetropolitan and metropolitan counties, and project served all ARC counties 
within a state (or served multiple counties in two or more states). Although 
Appalachia may be perceived as being largely rural, there are a number of 
metropolitan areas within the region. While rural and metropolitan regions of 
Appalachia have common problems, different approaches may be more or less 
successful in these areas. 

ARC grant size and total project cost. These variables are considered proxies for 
the size of a project. For project directors and grantees, project costs are one metric by 
which to determine whether or not a given project is feasible for the host organization 
and/or community. 

Years of ARC funding. Although all of the projects included here were funded 
during the 1990s, some projects received multiple years of funding. In some cases 
these may have been longer duration projects with the expected benefits to occur only 
after a period of capacity-building or development, while other multi-year projects 
may have been successful in their first year and received subsequent funding for 
continuation. The duration of a project is an important indicator of how long it takes 
to implement approaches, and how long it may take to realize participant outcomes. 
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Structure of the Report 



The remainder of this report provides the substantive findings from the evaluation. These 
results are organized as follows: 

■ Chapter 2 - Characteristics of the Study Sample 

■ Chapter 3 - Project Activities 

■ Chapter 4 - ARC Outcomes 

■ Chapter 5 - Project Sustainability and Expansion 

■ Chapter 6 - Summary and Recommendations 

■ Appendix A - Examples of Project’s ARC-fimded Outcomes 

■ Appendix B - Analysis of Outcomes Associated with the Overall Project 

■ Appendix C - Survey Data Disaggregated by Project Characteristics 

■ Appendix D - Notes on the Technical Approach 

■ Appendix E - Case Study Methodology, Findings, and Abstracts 

■ Appendix F - Project Descriptions 

■ Appendix G - Mail Survey 
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2. CHARACTERISTICS OF THE STUDY SAMPLE 



This chapter describes the characteristics of the 84 projects in the study sample. It begins 
with a portrayal of the communities that received educational support through the ARC, continues with a 
description of specific project characteristics, and provides an overview of the grant recipients and partner 
organizations that assumed responsibility for managing and implementing the ARC grants. Wherever 
possible, the characteristics of the study sample are compared with the full portfolio of education projects 1 
that received ARC funding during the 1990s. 2 

2.1 Community Characteristics 

ARC, in close collaboration with the 13 states of Appalachia, works to fund projects that will 
increase the knowledge and skill levels throughout the Region. In this section, we describe the 
communities that received educational services through an ARC education grant. Special emphasis is 
placed on the diversity of these communities and the types of economic and educational barriers that 
projects were designed to overcome. 



2.1.1 Project Scope 



The projects in the study sample were asked to indicate the geographic distribution of the 
individuals who were expected to benefit from their ARC grant. The majority of projects indicated that 
they were designed to serve residents in two or more counties (Figure 2-1). Specifically: 

■ One-third of the projects in the study sample were designed to reach individuals 
within a single county (30 percent) or a single city or town (4 percent). 

■ More than half were designed to provide services to multiple counties within a single 
state — i.e., in two or more adjacent counties (37 percent), in two or more nonadjacent 
counties (16 percent), or in all ARC counties in the state (2 percent). 

■ Just under one-tenth were designed to serve residents in two or more adjacent (8 
percent) or nonadjacent (1 percent) states. 



1 This excludes vocational education and workforce training projects, which will be examined in a subsequent evaluation* 

2 We are only able to make comparisons for project characteristics that are contained in the ARC database. We are not able to make comparisons 
for project characteristics that were derived from the mail survey. In addition, because the study sample and the full portfolio of education 
projects that received ARC funding were not independent of one another, we did not test whether similarities and differences between these two 
groups were statistically significant. 
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Figure 2-1. Geographic distribution of expected beneficiaries of ARC education projects (n=84) 
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NOTE: No projects reported a major metropolitan region as the geographic location of their project. 
SOURCE: 2000 mail survey of ARC grantees. 



Metropolitan Status. Almost half (48 percent) of the projects in the study sample were 



located in a nonmetropolitan area, compared to 44 percent of all ARC education projects (Figure 2-2). 
According to the descriptions that projects provided in their applications to ARC, these nonmetropolitan 
projects were often designed to serve residents who were living in severely isolated communities. For 
example: 



The target population will consist of individuals unable to access existing 
programs and employment due to isolation and lack of transportation. 
Allegheny County is sparsely populated with 51,742 people scattered 
across 1,047 square miles. (Linking Individuals to Fundamental 
Training) 



2.1.2 Metropolitan and Economic Status 
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Figure 2-2. Percent of projects by metropolitan status 
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NOTE: Percents may not sum to 100 due to rounding. 
SOURCE: ARC database. 



■ Macon County is a small, rural county located in northern Middle 
Tennessee with a population of approximately 14,000 residents. The 
primary employment in the county is factory work and agriculture, with a 
per capita income of $10,632. (Diversified Technology Program) 

Almost one-quarter (24 percent) provided services to a combination of nonmetropolitan and 
metropolitan communities, compared to 33 percent of all ARC education projects. Finally, 10 percent of 
the study’s projects were situated in metropolitan communities, and 19 percent provided services to all 
ARC counties within a state (or to multiple counties in two or more ARC states). Among all ARC 
education projects, 13 percent were serving metropolitan areas, and 10 percent were statewide or 
multistate. For these multicounty projects, the ARC database did not provide any indication of whether 
the educational services provided through the ARC grant were evenly distributed between the 
nonmetropolitan and metropolitan communities. Nor did it provide information as to whether the 
statewide/multistate projects targeted their efforts to metropolitan or nonmetropolitan areas. 

Economic Status. The Distressed Counties Program is designed to provide a mechanism for 
setting aside funds to the Region’s most impoverished communities. Begun in 1983, this program 
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provides a framework by which ARC can target a portion of its limited resources to those counties most 
in need of assistance. In addition, the program serves as a means by which the ARC can monitor the 
socioeconomic status of individual counties. Every year, the 406 counties of Appalachia are assigned to 
one of four economic categories: distressed, transitional, competitive, and attainment. Distressed counties 
are eligible for additional funding and lower matching requirements (20 percent). The match requirements 
are higher for transitional (50 percent) and competitive (80 percent) counties. Attainment counties are not 
eligible to receive funding from the ARC. 

To be classified as “distressed,” a county must have (1) a poverty rate that is at least 150 
percent higher than the national average, (2) a 3-year unemployment rate that is at least 150 percent 
higher than the national average, and (3) a per capita market income (i.e., per capita income less transfer 
payments) that is two-thirds or less than the national average. Counties with lesser degrees of distress are 
designated as transitional, and as counties become more prosperous, they move to competitive and then 
attainment status. At the time of the mail survey (FY 2000), 111 (27 percent) of Appalachia’s 406 
counties were classified as distressed, 65 percent as transitional, 5 percent as competitive, and 3 percent 
as attainment. 3 

In an effort to assess the extent to which the projects in the study sample were situated in the 
Region’s poorest communities, we calculated the proportion of initiatives that indicated they would be 
serving at least one distressed county in their application to the ARC. 4 Approximately, one-third (31 
percent) of the projects in the study sample were serving at least one distressed county, while 50 percent 
served no distressed counties, and 19 percent were statewide or multistate (Figure 2-3). 5 This finding 



3 In FY 200 1, the designations changed slightly, resulting in 1 14 distressed, 261 transitional, 22 competitive, and 9 attainment counties. 

4 As discussed earlier in this chapter, many of the projects in the study sample served more than one county. We therefore decided to focus on 
whether projects were identified as serving at least one distressed county in the ARC database. We were not, however, able to distinguish 
projects that were only providing services to distressed counties. In conducting this analysis, we used the county designation that was in place at 
the time the ARC grant was awarded — regardless of whether a given county’s economic designation had changed over time. Finally, there was 
no information in the ARC database as to whether some of the statewide or multistate projects targeted their efforts to distressed counties. 

5 A better way to examine these projects would be to assign a weighted average economic designation for statewide and multicounty projects. 
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Figure 2-3. Percent of projects by economic status 



Study Sample (n=84) 



All ARC Education Projects (n=320) 




H At Least one distressed county 
^ No distressed counties 



SOURCE: ARC database. 



□ Statewide or multistate 



suggests that the proportion of study sites serving at least one distressed county was slightly higher than 
the proportion of Appalachian counties designated as distressed in 2000, but similar to the proportion of 
all ARC education projects serving distressed counties. (We were not able to ascertain the number of 
projects in the study sample that were providing services in transitional counties. However, as is 
discussed in the next section, even projects situated in nondistressed counties tended to serve those 
community segments most in need of economic and educational support.) 

We also reviewed projects’ original requests for ARC support to gain a better understanding 
of the economic conditions they were designed to ameliorate. The following statements, pulled from these 
application materials, provide examples of the communities in which projects were situated: 

■ McCreary County has historically been one of the most disadvantaged 
counties in Kentucky. Designated as a “distressed county” by ARC, 

McCreary County faced a 1990 poverty rate of 46 percent. (McCreary 
County Learning Center Expansion Project) 

■ Breathitt County has a total population of 14,905 with 4,209 of its 
residents being children under the age of 18. Forty-six percent of 
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children live below the poverty level. (Montessori Preschool Relocation 
and Expansion Project) 

The economic and education blight of Columbiana County is reflected in 
the following statistics: (1) nine of the eleven districts ranked in the 
bottom 30 percent in annual income and the bottom 15 percent in 
property valuation per pupil, and (2) nine of the eleven districts ranked in 
the bottom 200 districts in the State in terms of overall wealth. 
(Columbiana County Business/Education Partnership for Interactive TV 
Technology Labs) 



2.1.3 Extent to Which Projects Served Disadvantaged or Underserved Community Segments 

Our analysis of the economic designations of counties in the study sample provides a useful 
indicator of whether study sites were focusing on the Region’s most distressed communities. It does not, 
however, address whether projects in nondistressed counties are focusing their efforts on those 
community segments most in need of economic and educational assistance. To address this issue, survey 
respondents were asked if their projects were designed to serve members of their communities who were 
traditionally disadvantaged or underserved. Almost all (81 of the 82 projects that responded to this survey 
item) indicated that their project was designed to address at least one disadvantaged/underserved 
community segment — and the average project was designed to address five disadvantaged/underserved 
community segments. The most commonly targeted community segments were individuals who were 
impoverished (82 percent) and/or geographically isolated (77 percent). In addition, well over half of 
projects were designed to target those who were unemployed or underemployed (70 percent), school 
dropouts (60 percent), disabled (59 percent), or illiterate (57 percent) (Table 2-1). 
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Table 2-1. Number and percent of ARC projects that indicated they intended to use 

project-related services or resources to otherwise benefit various disadvantaged 
or underserved community segments (n=82) 



Characteristic of underserved group 



Number of projects 



Percent of projects 



Extreme poverty 

Geographically isolated 

Unemployed/underemployed 

School dropouts 

Disabled 

Illiterate 

Limited English speaking 

Migrant workers/students 

Other groups 



NOTE: Two projects did not provide any information 
SOURCE: 2000 mail survey of ARC grantees. 



67 


82 


63 


77 


57 


70 


49 


60 


48 


59 


47 


57 


29 


35 


29 


35 


16 


20 



on the intended community segments to be served. 



Problems or Barriers Projects Were Designed to Address. The document review and site 
visits provided additional evidence that the projects in the study sample were targeting community 
segments most in need of educational support. These reviews also suggest that projects often attempted to 
ameliorate multiple barriers. For example, many of the adult learner projects designed to reduce 
intergenerational illiteracy also sought to address other issues — e.g., the lack of a high school diploma or 
unemployment/underemployment. Similarly, projects serving elementary/secondary students commonly 
addressed a combination of issues — e.g., a lack of educational technology or lab equipment, undertrained 
teachers, rising high school dropout rates. The following examples, pulled from applications to the ARC 
and case study reports, illustrate the types of barriers faced by projects’ intended beneficiaries: 

■ Eastern and southern Kentucky is a rural, isolated region that has been 
economically dependent of the coal mining industry for years. Little 
education was needed to be a coal miner, and during the boom years the 
pay was very good, sometimes as high as $30,000 for entry-level 
positions. Hundreds of young men saw no need to obtain an education 
when they could go into the mines and earn that salary. Indeed, their 
fathers and grandfathers had done it before them. (Local Affiliate 
Expansion Project application) 

■ Families live in neighborhoods where education is not seen as a high 
priority or as something realistically attainable. Educational nurturing is 
not an accepted value for parents who have not completed high school, 
who may be illiterate themselves, and who live under the stressful 
conditions of low wage employment... Feelings of inferiority, cultural 
differences, perceived unfriendliness of teachers, hostility toward school 
for perceived wrongs, and limited communication and problem-solving 
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skills are the norm... For severely dysfunctional families, there may be 
problems with drugs and alcohol abuse, absence of parental supervision 
and protection of children, nonacceptance of traditional standards for 
hygiene and sanitation, impoverished living conditions, family violence, 
and adult illiteracy. Communication between the parent and child is 
usually poor and there is little or no parental encouragement and support. 
(School Outreach Project application) 

Teenage pregnancy (often compounded over several generations) as well 
as the increasing disintegration of extended families and communities 
has resulted in the diminishing of parenting knowledge passed from 
generation to generation among many of eastern Tennessee’s poor. A 
host of parenting deficiencies has resulted from this state of affairs, such 
as poor hygiene (unclean bottles, head lice among children), child abuse, 
and a lack of understanding of child development. In addition, because of 
the fragmentation of communities, many parents do not have knowledge 
of or access to resources available to them. Many parents in eastern 
Tennessee are forced to raise their children under extremely difficult 
circumstances with few external supports. Further, these children fall 
behind in their development to such an extent that they may never fully 
recover. Many enter school lacking the fundamental cognitive, linguistic, 
social, and physical skills necessary for success. (Partnering with 
Parents case study) 



Another common condition that projects described in their applications was an economic or 
geographic barrier that needed to be overcome. These barriers typically worked together in synergistic 
ways, creating a difficult cycle to break. For example, geographic barriers such as isolation create 
problems in their own right, but feed into other problems as well, such as the provision of adequate 
services and the ability to attract business and industry that would support the local economy. (See 
Exhibit 2-1 for examples of economic and geographic barriers that projects faced.) 6 



Finally, many of the projects described how their efforts would address a community’s lack 
of access to a critical service (e.g., transportation, affordable day care) or educational opportunity (e.g., 
museum, computer lab, science and technology equipment for students). In some of the most 
economically or geographically isolated communities, providing even basic services (let alone developing 
models of advanced service delivery) sometimes proves difficult. Exhibit 2-2 illustrates the types of 
access barriers that two case study sites were designed to overcome. 



fi The exhibits on the following pages and throughout the report provide specific examples drawn from documents to provide a richer picture of 
issues as appropriate. 
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Exhibit 2-1. Examples of economic or geographic problems 
barriers that projects were designed to address 



■ The David School: Success Bound. Public education in this region has been distressed primarily 
due to an unstable economy and an inadequate tax base. These factors have caused overcrowding 
and a lack of funding to develop appropriate programming to the special needs of a large portion 
of mountain youth. The dropout rate in the immediate three-county area has reached epidemic 
proportions. Between 45 and 55 percent of students entering 9 th grade do not complete 12 th grade, 
as reported by the Kentucky Department of Education “Holding Power Report.” 

■ Infrastructure Development for the 21st Century. Nowhere in Virginia are the effects of 
disparity of educational opportunity more keenly felt than in the counties of far southwest 
Virginia, where school divisions are forced to eliminate course offerings, freeze hiring of 
instructional staff or reduce the number of teachers, and even close schools. Elsewhere, wealthier 
school divisions continue to spend up to 2 l A times more per pupil than less wealthy school 
divisions located chiefly in the rural areas of the state. The result is that poorer areas have fewer 
curricular offerings, less state-of-the-art technology, more poorly equipped labs, and higher 
student-to-teacher ratios than schools divisions in wealthier areas. 

■ Thanks A Million. In West Virginia, 570,857 (or 44 percent) of the adults over the age of 18 
have not completed high school. Although grade level completion is not an entirely accurate 
account of functional literacy, this high percentage does indicate that the chances of a major basic 
skills problem could certainly be a significant one. With escalating workplace requirements, the 
economic impact of functional illiteracy carries a high price. Despite the valiant efforts of 
organizations such as public education, libraries, and volunteer literacy programs, less than 5 
percent of the target population is being served. Limited funding for adult literacy programs has 
put strain on providing services adequate to reach and teach the large target population. In some 
counties, adults in need of services are put on waiting lists because classes are overcrowded or no 
volunteer tutors are available. Limited budgets often force programs to cut back on the number of 
class sites within the county. As a result, adult learners are left with the option of driving to a 
class on the opposite side of the county, often over winding, mountainous roads, or simply 
forgetting about pursuing their education. 

SOURCE: Document review of ARC education projects. 
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Exhibit 2-2. Examples of access barriers that case study projects were designed to address 



■ Science Center of West Virginia. Since its inception, the Science Center of West Virginia has 
been the only facility in this area of the state that offered hands-on science education 
opportunities. Sunrise Science Hall, located in Charleston, also began offering some informal 
science education opportunities along with its original art program about that time, but that 
facility is located some 2 hours north of Bluefield — and largely out of reach of many of the area’s 
schools in terms of both travel time and expense. Sunrise is also a more static viewing 
environment, with few interactive opportunities for visitors. Another problem facing the region 
was a lack of resources and equipment in all schools to adequately provide students with the 50 
percent level of hands-on instructional time required by the state’s 1993 science standards. At the 
time the ARC grant proposal was submitted, Mercer County and its neighboring counties had 
limited science resources available. The elementary education coordinator for the county 
indicated that this continues to be an issue, but the situation had improved somewhat at the 
middle and high school levels. Mercer County had begun to consolidate its schools over the past 
few years due to enrollment decreases. This consolidation has allowed for an increase in 
laboratory facilities and equipment at the remaining secondary schools. The elementary schools, 
however, have not experienced any real change in this regard. For example, with school funding 
being allocated on a per-student basis, some of the smaller schools, which have as few as 90 
students, do not have excess funds to spend on science equipment. Recognizing the dual 
necessities of increasing students’ accessibility to informal science education and increasing 
science resource availability to schools in the region, the founder/president of the Science Center, 
who then was also president of the Alliance, began raising money to create hands-on exhibits and 
an active outreach program. 

■ Science and Math To Go! (SMTG!). The hands-on, discovery-oriented experiences supported 
by SMTG! were often not available for a variety of reasons: (1) lack of time by teachers to seek 
and purchase supplies, (2) general lack of space and equipment to prepare hands-on laboratory 
experiences, (3) general lack of equipment for student use, (4) lack of availability to store and 
maintain equipment, and (5) lack of funds to purchase supplies and equipment. Additionally, 
project staff noted that many elementary teachers have little preparation in science content 
knowledge and pedagogy. These teachers are frequently uncomfortable with any science beyond 
a textbook or simply do not know what to do. Moreover, the schools and districts face an 
inequitable distribution of quality teaching materials — i.e., not all students have equal access to 
quality science content. 

SOURCE: Document review of ARC education projects. 
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2.2 



Project Characteristics 



This section provides information on the projects themselves. It discusses project types, 
funding sources, amount of ARC funding and total project costs, relationship of the ARC grants to other 
project activities, and duration of the projects. 



2.2.1 Project Type 



Over three-quarters of the projects in the study sample fell into four of nine categories that 
the ARC uses to classify its education grants, including adult literacy (27 percent), secondary education 
(24 percent), basic skills (14 percent), and math/science education (11 percent) (Table 2-2). The 
remainder were distance learning, dropout prevention, preschool, elementary education, and educational 
partnership projects. 



Table 2-2. Percent of projects by ARC education category 



ARC education category 


Percent of projects in the 
study sample 
(n=84) 


Percent of all ARC 
education projects 
(n=320) 


Adult literacy 


27 


17 


Secondary education 


24 


21 


Basic skills 


14 


7 


Math/science education 


11 


4 


Distance learning 


6 


13 


Dropout prevention 


6 


5 


Preschool 


5 


4 


Educational partnership 


4 


3 


Elementary education 


4 


3 


Childcare 


0 


9 


Telecommunications 


0 


6 


Technical assistance 


0 


4 


Teacher training 


0 


2 


Libraries 


0 


1 



NOTE: Percents may not sum to 100 due to rounding. 
SOURCE: ARC database. 



The distribution of study sample categories is roughly similar to the distribution for all ARC 
education projects, with the exception of adult literacy (27 percent of study sample projects, but only 17 
percent of all projects), basic skills (14 and 7 percent, respectively), math/science education (1 1 and 4 
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percent, respectively), and distance learning (6 and 13 percent, respectively). In addition, several 
categories represented among the full portfolio of ARC education projects were not included in the study 
sample, including childcare (9 percent of all projects) and telecommunications (6 percent). 



In conducting the document review, we determined that there was considerable overlap in 
these categories — with many initiatives providing services or addressing barriers in two or more ARC 
education categories. For example, a project designed to provide a dropout prevention program to 
secondary education students could be classified as “dropout prevention” or “secondary education.” 
Therefore, for the purposes of the study, projects were reclassified according to the types of individuals 
who received services or otherwise benefited through the ARC grant. As shown in Table 2-3, these five 
project types include: 

■ Preschool age. This category, accounting for 7 percent of projects in the study 
sample, includes the six initiatives that provided services to preschoolers (or parents 
of preschoolers). 

■ Elementary/secondary (all students). This category, accounting for 38 percent of 
projects, comprises initiatives designed to benefit all students — i.e., no effort was 
made to withhold or target services to a subset of students. Thus, projects designed to 
provide enhanced curriculum, math/science education, or enhanced technology access 
to all students were included in this category. 

■ Elementary/secondary (targeted students). This category, accounting for 10 percent 
of projects, includes the eight initiatives that targeted their services to a specific subset 
of youth — e.g., students at risk of dropping out of public school, students who have 
already dropped out, students with severe learning disabilities. 

■ Adults. This category, accounting for 33 percent of projects, includes initiatives that 
primarily provided services to adults — e.g., literacy training, basic skills, parenting 
skills, GED preparation. 

■ Communitywide. The final category, accounting for 12 percent of projects, includes 
the 1 0 projects that provided services to entire communities — e.g., distance learning 
centers, support and technical assistance to community-based organizations, career 
development centers. 
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Table 2-3. Number and percent of projects, by intended beneficiary of the project (n=84) 



Project type 



Percent of projects 



Preschool age 7 

Elementary /secondary, all students 38 

Elementary/secondary, targeted students 10 

Adults 33 

Communitywide 12 



SOURCE: Document review of ARC grantees. 



It should be noted that while care was taken to assure that projects were assigned to an 
appropriate category, there was still considerable opportunity for overlap. For example, a number of 
projects that were primarily designed to benefit youth were also providing limited services to adults, or a 
school-based project might have been actively encouraging parents to enroll in a GED course. In such 
cases, a project’s primary intended beneficiary was used to make an assignment to one of the five 
categories. (A comparison to all ARC education projects cannot be made.) 



2.2.2 Funding Source 

The projects in the study sample received ARC funding in one of two ways — through their 
individual state or through the Commission. 7 Seventeen (20 percent) of the study sites were funded 
through the Commission, with the remaining 67 projects being funded through their respective states. 

Two states — Georgia and Kentucky — each had 17 state-funded projects (20 percent of the 
study sample). Other states had relatively fewer projects in the sample. For example, New York had eight 
projects, Ohio had five, and Mississippi and Tennessee each had four. As shown in Table 2-4, Georgia 
and Kentucky were overrepresented in the study sample, while New York was underrepresented. 



7 Commission- funded projects are designated as such because the funding for the project comes from the ARC set-asides, as opposed to an 
individual state’s allocation of money. Commission projects may operate in a single state or in multiple states. 
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Table 2-4. Percent of projects by the project’s source of funding 





Percent of projects in the 


Percent of all ARC 


Source of funding 


study sample 


education projects 




(n=84) 


(n=320) 



Alabama 1 6 

Georgia 20 12 

Kentucky 20 9 

Maryland 0 2 

Mississippi 5 3 

North Carolina 4 3 

New York 10 17 

Ohio 6 8 

Pennsylvania 2 4 

South Carolina 2 1 

Tennessee 5 3 

Virginia 1 3 

West Virginia 4 5 

Commission 20 23 



NOTE: Percents may not sum to 100 due to rounding. 
SOURCE: ARC Database. 



2.2.3 Funding Amount and Total Project Cost 

By design, ARC grants are often awarded in conjunction with funds from other sources — 
e.g., local businesses, other federal or state agencies, donations from private foundations or other sources. 
The ARC requires that projects provide a match of between 20 percent and 80 percent, and this amount 
varies by the economic designation of the community within which the project is located. Projects in 
distressed counties are expected to match only 20 percent of the ARC grant (compared to projects in 
competitive counties where their match is 80 percent of the grant amount). In the case where a project is 
located in counties with various designations, the project is assigned a weighted average score in order to 
determine the required match amount. Indeed, many projects had funding from other agencies over and 
above their ARC-required match. 

ARC Grant Amount. The size of the ARC grant varied tremendously across the study 
sample — from a low of $14,000* to a high of $3 million. Half of the projects received less than $100,000 
in ARC funding — with 21 percent receiving $50,000 or less and 30 percent receiving from $50,001 to 



X 



Projects receiving less than $10,000 were excluded from the study sample and the universe of all ARC education projects. 
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$100,000 (not shown in tables). 9 Almost one-third (32 percent) of projects received between $100,001 
and $200,000 — with the remaining 17 percent receiving over $200,000. The distribution among all ARC 
education projects was roughly similar. Specifically, 34 percent received $50,000 or less, 27 percent 
received from $50,001 to $100,000, 25 percent received between $100,001 and $200,000, and 14 percent 
received over $200,000. 

Total Project Cost. Projects also varied in terms of the total financial resources that they 
had at their disposal — from a low of $30,000 to a high of $4,960,000. While 25 percent of the projects 
had total expenditures of $100,000 or less, 10 percent had expenditures of over $900,000. The remaining 
projects reported expenditures between $100,001 and $200,000 (33 percent) or between $200,001 and 
$900,000 (32 percent). As expected, there was a direct relationship between the size of the ARC grant and 
total project cost (not shown in tables). Total costs among all ARC education projects matched that of the 
study sample with slight differences in the lowest category (34 percent) and the highest category (5 
percent). 



2.2.4 Project Duration 

The average project in the study sample received 1 to 2 years of ARC funding, with 26 
percent receiving ARC funding for a single year, 41 percent receiving ARC funding for 2 years, and 33 
percent receiving funding for 3 or more years (not shown in tables). 

As discussed previously, the study sample comprised projects that received an ARC 
educational grant during the 1990s. 10 However, as shown in Figure 2-4, there was some variation with 
respect to the specific years in which projects reported receiving financial support. Projects were most 
likely to indicate that they received financial support from ARC in the mid-1990s — e.g., 33 percent 
indicated 1995 as a year in which they were funded by ARC. 



9 Funding amounts were combined in cases where projects received multiple years of financial support through the ARC. 

111 We excluded from our sample projects that received initial funding in 1999. In addition, the survey and document review did not obtain 
information about activities that occurred prior to 1990, even if a project had received financial support through ARC in the 1980s. 




2-15 



43 



Figure 2-4. Percent of projects that reported receiving ARC funding during the years 1990-99 
(n=81) 




0 20 40 60 80 100 



Percent of projects 



NOTE: Three projects did not provide any information about the years of funding. 
SOURCE: 2000 mail survey of ARC grantees. 



It is likely that this trend reflects the way in which the study sample was selected. 
Specifically, projects funded in the early part of the decade were less likely to have a contact person who 
was knowledgeable about the project and therefore able to respond to the mail survey. (As discussed in 
Chapter 1, projects that appeared to lack a knowledgeable respondent for the mail survey were excluded 
from the study.) In addition, an effort was made to select those projects that had already completed their 
ARC grant. As such, only 14 percent of the projects in the study sample reported that they had been 
funded as far back as 1990, and only 1 1 percent had received funding in 1999. 
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2.3 



Grant Recipient Characteristics 



This section provides information about the organizations responsible for overseeing and 
implementing the ARC educational grants. It includes a description of the entities that served as grant 
recipients and project partners — with particular emphasis on the contributions of project partners to the 
overall effort. 



2.3.1 Characteristics of Grant Recipients 

Although all of the projects in the mail survey had education-related goals, not all originated 
from educational organizations. Indeed, projects were overseen by a wide range of organizations, 
including education entities (63 percent), other organizations (26 percent), and government agencies (1 1 
percent) (Table 2-5). 



Table 2-5. Number and percent of ARC education projects, by the type of organization of the 
grant recipient (n = 84) 



Organization type 



Number of projects 



Percent of projects 



Education organizations 

Early childhood education organization. 

K-12 school or school system 

Higher education institution or consortium. 

Adult education organization 

Local education agency 

Other education entity 

Other organizations 

Social service agency 

Library 

Museum or other entity 

Community development organization 

Professional organization/union 

Public broadcasting station 

Religious organization 

Health care organization 

Public safety organization 

Nonprofit organization 

Other community organization 

Government organizations 

State government agency 

County government agency 

City or municipal government agency 

Tribal government 

Other government entity 



53 


63 


3 


4 


17 


20 


3 


4 


9 


11 


6 


7 


15 


18 


22 


26 


0 


0 


1 


1 


2 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


18 


21 


1 


1 


9 


11 


6 


7 


1 


1 


1 


1 


0 


0 


1 


1 



NOTE: Details may not sum to totals due to rounding. 
SOURCE: 2000 mail survey of ARC grantees. 
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The most common education organizations receiving an ARC grant were K-12 schools or 
school systems (17 grant recipients), adult education organizations (9 grant recipients), and local 
education agencies (6 grant recipients). Additionally, 15 other types of education organizations received 
grants, including a consortium of education organizations, a regional educational laboratory, and a joint 
vocational school. The following examples, culled from the document review and case studies, illustrate 
the range of educational organizations that served as grant recipients: 

■ Towns County Middle School was part of the Towns County 

Comprehensive School complex. This complex also housed a preschool, 
kindergarten, elementary, and high school classrooms. Prior to receiving 
the ARC grant, the school had already taken steps to begin providing 
access to learning technologies and was interested in expanding access to 
all students, families, and faculty. (Mobile Technology Project) 

■ The David School was a private, nondenominational nonprofit 

multifaceted educational service network. Its core curriculum, a full-time 
alternative secondary academic-vocational agenda, was established in 
response to the critical problem of students in the mountains of eastern 
Kentucky dropping out of school. (The David School: Success Bound) 

■ Science and Math To Go! was a curriculum and professional 

development program run out of the Anderson Oconee Pickens Hub at 
Clemson University. The hub was one of 13 across the state of South 
Carolina established in the planning for and receipt of a National Science 
Foundation grant for systemic education reform. Hubs served as math 
and science resource centers for the districts they encompass. (Science 
and Math To Go!) 

Most of the government agencies that served as grant recipients were at the state level. Only 
one county and one city agency were so designated. In addition, one project was run through a national 
laboratory at the Department of Energy. Examples of government agencies that served as grant recipients 
follow: 



The Allegheny County Employment and Training Center was a 
composite of agencies whose purpose was to assist individuals in 
achieving employment goals and self-sufficiency. Eight agencies 
operated cooperatively in the center. (Linking Individuals to 
Fundamental Training) 

The Kentucky Literacy Commission was a specific agency designed to 
serve adult nonreaders with a general fund appropriation to ‘"begin” 
program development. The Commission served adults age 16 and over 
who read below a sixth grade level through its network of local literacy 
programs in all of Kentucky’s 120 counties. (Operation Read) 
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The remaining 22 grant recipients represented other agencies — i.e., nonprofit organizations 
(18 projects), museums (2 projects), library (1 project), and for-profit business (1 project). Examples of 
other entities that served as grant recipients are provided below; 



Thanks A Million Foundation (TAM) represented a cooperative effort of 
the public and private sectors to promote and support existing and new, 
innovative adult literacy efforts. (Thanks A Million) 

Forward in the Fifth was a nonprofit organization dedicated to improving 
the educational attainment of citizens in 39 counties of eastern and 
southern Kentucky. (Forward in the Fifth Math and Science Project) 



2.3.2 Characteristics of Project Partners 

While ARC grants are awarded to individual organizations, the implementation of most 
projects involved collaborations between the grant recipient and a number of partnering entities. 
Partnerships may be formed between the recipient and a wide range of organizations, including 
government agencies (local, state, or federal), education agencies, private foundations or companies, 
service providers, and other agencies. 

Survey respondents were provided a list of nine different types of partners and asked to 
indicate how many partners of each type were involved with their projects. The 77 projects that responded 
to this item reported a total of 671 partners (Table 2-6). In addition; 

■ Sixty-one percent of projects reported that at least one of their partners was a local 
education agency. 

■ Forty percent reported that at least one of their partners was a private sector company. 

■ Thirty-nine percent of projects reported having a state education agency as a partner. 




^2 
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Table 2-6. Total number of partners involved and those that indicated having partnerships 

with each type of organization (n=77) 



Type of organization 



Local education agency 

Private sector company 

Other organization 

Local government (county/town) agency. 

Private foundation 

Nonprofit organization 

State education agency 

Federal agency 

State agency (excluding education) 



Total number of partners 
(n=671) 


Percent of projects 


187 


61 


182 


40 


62 


10 


60 


25 


52 


30 


46 


35 


39 


39 


27 


33 


16 


17 



NOTE: Seven projects did not provide any information about the type of partnerships for the ARC project Percents do not sum to 1 00 
because many projects had partnerships with more than one organization type. 

SOURCE: 2000 mail survey of ARC grantees. 



Partners’ most prevalent contribution came in the form of financial support — with 75 percent 
of projects indicating that at least one of their partners provided monetary assistance (Table 2-7). Other 
contributions commonly provided by partners included space or facilities (cited by 68 percent of 
projects), in-kind support (67 percent), personnel (58 percent), and equipment or equipment discounts 
(41 percent). The following examples, taken from the case studies, illustrate how these partnerships 
contributed to project success: 

■ The Science Center developed a close relationship with the National 
Aeronautics and Space Administration (NASA), especially the Langley 
Research Center in Hampton, Virginia. For example, NASA personnel 
and exhibit materials make routine visits to the Science Center to provide 
professional development workshops to area teachers and informal 
science education opportunities to students and other area residents. 

These have included an exhibit of moon rocks, a $240 million lunar 
materials disk (soil and mineral samples), and a rocket that NASA 
transported to the center for display. (Science Center of West Virginia) 

■ Towns County Middle School achieved its match by soliciting funds 
from several local businesses to purchase individual laptop computers. 

According to the principal, this approach helped to create community 
awareness about the project. The need to increase community awareness 
of— and support for — the Mobile Technology Project was critical for two 
reasons. First, a number of the local businesses were owned or operated 
by parents whose children would be using the laptops at home. As such, 
this was viewed as another strategy for increasing parents’ involvement 
in their children’s schoolwork. Second, the project was viewed as an 
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opportunity to reinforce the relationship between the school and the local 
businesses that would likely employ many of the community’s high 
school and college graduates. For example, a representative from a local 
bank that contributed funds for the project visited the school to 
demonstrate the electronic spreadsheets he routinely used at work. 
(Mobile Technology Project) 

■ Science and Math To Go! (SMTG!) nurtured a number of linkages with 
local and South Carolina-based corporations. The initial contribution 
these partnership was generally financial — e.g., Duke Power, Michelin, 
DuPont, and BMW all provided monetary support to SMTG! Over time, 
the name recognition associated with these entities has made it easier for 
SMTG! to obtain support from local school boards. (Science and Math 
To Go!) 



Table 2-7. Total number of contributions and percent of projects that reported partner 



organizations provided contributions to the ARC project in the following ways 
(n=79) 




Total number of 




Type of contribution 


contributions 

(n=952) 


Percent of projects 


Provided cash 


354 


75 


Provided space or facilities 


158 


68 


Provided in-kind services 


220 


67 


Provided personnel 


120 


58 


Provided equipment or equipment discounts 


66 


41 


Provided other contributions 


34 


6 



NOTE: Five projects did not provide any information about the type of service or resource provided by partner organizations. Percents do not 
sum to 1 00 because partners often provided more than one contribution. 

SOURCE: 2000 mail survey of ARC grantees. 



2.4 Summary 

According to its Strategic Plan, part of ARC’S mandate is to “address economic 
underdevelopment that is deeply rooted in regionwide, historic patterns of isolation, exploitation, and 
outmigration.” Study findings suggest that the project in the study sample contributed to the 
Commission’s goal of reaching those segments of Appalachia that are rural and geographically isolated. 
Most projects provided services to rural areas — with almost half being situated in a region that was 
entirely nonmetropolitan. Equally significant, the community segments that were targeted generally 
reflected the ARC’S goal (as articulated in both its Strategic Plan and the provisions of its Distressed 
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County Program) of serving those most in need — including persons in extreme poverty, persons who are 
geographically isolated, those who are unemployed or underemployed, youth at risk of dropping out of 
school, and persons with disabilities. In addition, one-third of projects were serving areas that included 
distressed counties. Although half of the projects in the study sample did not serve a distressed county, 11 
their efforts were still focused on those community segments most in need of services. Also, it is likely 
that many residents in nondistressed areas faced barriers similar to those in distressed areas and therefore 
required similar assistance from ARC. 

The projects in the study sample primarily focused on elementary and secondary students 
and adults, with small percentages serving preschool students and communities as a whole. Among 
projects serving elementary and secondary students, the vast majority were addressing the needs of all 
students, rather than of a targeted group of students. 

A comparison of projects in the study sample to the universe of all ARC projects funded 
during the same period found that the sample and the universe were roughly similar in economic and 
metropolitan status, ARC grant amount, and total project cost. All ARC states except Maryland (only 2 
percent of all education projects) were included in the study sample. Georgia and Kentucky were slightly 
overrepresented in the sample, while New York was slightly underrepresented. The mix of ARC project 
types in the sample varied only slightly from those of all education projects. Adult literacy, basic skills, 
and math/science education were overrepresented in the study sample, while distance learning was 
underrepresented. Child care and telecommunications were absent from the sample altogether. Finally, we 
were unable to match project types used for the study with those of all education projects due to the 
reclassification of the sample into adult, elementary/secondary (all students), elementary/secondary 
(targeted students), preschool, and communitywide. 



1 1 The remainder of projects were statewide or multistate and were not assigned an economic designation. 
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3. PROJECT ACTIVITIES 



Residents of Appalachia commonly encounter a broad range of geographic, economic, and 
educational barriers. It is therefore not surprising that the projects in the study sample were primarily 
designed to address a prevalent condition (e.g., chronic unemployment, excessive school dropout rates, 
insufficient educational and technical skills) and/or a lack of access to educational resources or facilities. 
This chapter addresses the range of strategies that were used to overcome these barriers. Of particular 
interest is whether projects were able to conduct their activities as planned — and the factors that affected 
the degree to which these activities were successfully implemented. 



3.1 Educational Activities Conducted by Projects in the Study Sample 

The projects in the study sample employed a wide variety of approaches designed to 
overcome the Region’s geographic, economic, and educational barriers. To determine which activities 
were used most prominently, the mail survey provided respondents with a list of 25 activities across the 
following six categories: improve physical plant, enhance telecommunications applications, develop 
educational resources, provide training to students and other community residents, provide support 
services, and provide community outreach. 1 Respondents were also provided an opportunity to describe 
“other” activities not included on the list of survey options. 

Several important caveats should be considered when reviewing this discussion of project 
activities. First, as noted previously, many of the projects in the sample had received funds from ARC 
(including the grant match), as well as other local, state, and/or federal sources. In such cases, the 
project’s implementation of activities relied on a combination of funding streams, with some activities 
funded solely by ARC, some funded solely by other sources, and some funded through a combination of 
sources. For example, a project may have used ARC funds to purchase computers, but funds from other 
sources to employ teachers and develop training materials for students. As a result of this combination of 
activities, students may have had increased access to test-preparation materials and scored higher on 
college entrance exams, resulting in greater acceptance rates to college. While it is possible for us to 
disaggregate activities funded by ARC from those funded by other sources (and, in fact, we have), it is 



Six projects that responded to the mail survey did not provide any data regarding the types of activities they conducted. These six projects were 
excluded from the analyses conducted for this section of the chapter. 



not possible to link specific activities (funded by ARC or otherwise) with the outcomes discussed in the 
next chapter. Specifically, it is not possible to attribute a project’s outcomes, which result from the totality 
of its activities, to any specific activity. 

A second caveat is that the options on the survey ranged from discretely defined tasks (e.g., 
install science lab) to broader all encompassing ventures (e.g., establish partnerships). The activities also 
varied with respect to how expensive and/or difficult they were to implement — e.g., installing computers 
is potentially cheaper and easier than developing a distance education network. Thus, although the 
discussion throughout this chapter regards all undertakings as being equal, the implementation of each 
activity actually required a unique combination of skills, costs, staffing requirements, community 
partnerships, and level of effort. 



3.1.1 Distribution of Activities 

The 78 projects that responded to this survey item reported conducting a total of 682 distinct 
activities. Of this number, 556 (82 percent) were partially or fully funded by ARC. 2 The majority of projects 
reported an ARC-funded activity aimed at developing educational resources (78 percent of projects), providing 
training to students and other community residents (71 percent), and/or installing/enhancing 
telecommunications applications (64 percent) (Table 3-1). Less than one-third of projects reported providing 
support services (28 percent) or physical plant (27 percent) activity. 

Across all activity types, the most frequently cited ARC-funded undertakings were installing 
computers (51 percent of projects), providing academic skills training (46 percent), providing computer 
training (45 percent), providing teacher or tutor training (45 percent), and providing literacy training 
(37 percent) (Table 3-2). The least frequently reported ARC- funded activities were renovating structures 
(7 percent of projects), building new structures (7 percent), providing emotional or psychological 
counseling (6 percent), and installing a science lab (5 percent). 



2 As noted in Chapter 2, ARC was the primary source of funding for some projects and a secondary source of funding for others. The mail survey 
was used to obtain information on all project-related activities regardless of their funding source. For each activity that a project conducted, the 
mail survey also obtained information on whether it was covered under the ARC grant. Therefore, in the discussion of project activities, a 
distinction is made between all activities reported by a project and those activities that were at least partially underwritten by ARC (i.e., ARC- 
funded activities). 





Table 3-1. Percent of projects reporting at least one ARC-funded activity in a given category, by 
project characteristics 









Category of activity 






Project characteristic 


Education 

resources 


Training 


Telecom- 

munica- 

tions 


Commu- 

nity 

outreach 


Support 

services 


Physical 

plant 


All projects (n=78) 


78 


71 


64 


42 


28 


27 


Project type 


Preschool age (n=6) 


83 


67 


17 


67 


67 


0 


Elementary/secondary, all students (n=31) 


94 


58 


74 


39 


19 


32 


Elementary/secondary, targeted students (n=7) 


86 


86 


14 


57 


71 


14 


Adults (n=25) 


60 


92 


72 


32 


24 


24 


Communitywide (n=9) 


67 


44 


78 


56 


11 


44 


Project scope 


Single town or county (n=25) 


64 


76 


88 


36 


32 


40 


Adjacent counties (n=30) 


77 


67 


70 


43 


20 


23 


Nonadjacent counties (n=23) 


96 


70 


48 


48 


35 


17 


Economic status 


At least one distressed county (n=24) 


79 


75 


50 


38 


29 


21 


No distressed counties (n=38) 


71 


74 


84 


42 


29 


40 


Statewide or multistate (n=16) 


94 


56 


38 


50 


25 


6 


Metropolitan status 


Nonmetropolitan only (n=38) 


74 


82 


79 


40 


34 


34 


Metropolitan only (n=6) 


50 


83 


83 


67 


17 


33 


Both metro and nonmetro (n— 1 8) 


83 


56 


50 


33 


22 


28 


Statewide or multistate (n=16) 


94 


56 


38 


50 


25 


6 


ARC grant size 


Less than $50,000 (n=17) 


77 


71 


53 


18 


12 


0 


$50,001 -$100,000 (n=22) 


73 


77 


68 


46 


46 


50 


$100,001 -$200,000 (n=25) 


80 


72 


68 


52 


28 


20 


More than $200,000 (n=14) 


86 


57 


64 


50 


21 


36 


Total project cost 


Less than $100,000 (n=18) 


78 


72 


50 


33 


28 


17 


$100,001 -$200,000 (n=27) 


74 


74 


70 


44 


33 


33 


$200,001 -$900,000 (n=25) 


84 


68 


68 


48 


28 


24 


More than $900,000 (n=8) 


75 


63 


63 


38 


13 


38 


Years of ARC funding 


1 year (n=21) 


91 


67 


71 


38 


24 


29 


2 years (n=33) 


82 


64 


64 


42 


21 


30 


3 or more years (n=24) 


63 


83 


58 


46 


42 


21 



NOTE: Six projects did not provide any information about the types of activities their projects were designed to conduct 
SOURCE: 2000 mail survey of ARC grantees. 
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Table 3-2. Percent of projects that reported various activities and percent of projects that 
reported various activities were fully or partially funded through ARC (n=78) 



Type of activity 


Percent of projects 


Percent of projects with 
ARC-funded activities 


Physical plant 






Install/replace mechanical equipment 


20 


17 


Renovate structures 


14 


7 


Build new structures 


10 


7 


Other physical plant activity 


4 


4 


T elecommunications 


Install computers 


56 


51 


Install/develop network 


32 


26 


Develop distance education system 


19 


16 


Other telecommunications activity 


6 


5 


Educational resources 


Install science lab 


11 


5 


Install other special use classroom 


26 


24 


Develop computer-based educational materials 


32 


25 


Develop paper-based educational materials 


33 


26 


Develop teacher training program/materials 


44 


32 


Provide teacher/tutor training 


63 


45 


Other educational resources activity 


20 


17 


Training 


Provide literacy training 


44 


37 


Provide computer training 


56 


45 


Provide GED preparation training 


36 


30 


Provide job skills training 


47 


35 


Provide parenting skills training 


29 


23 


Provide academic skills training 


63 


46 


Provide peer tutoring 


22 


17 


Other training activity 


6 


6 


Support services 


Provide emotional or psychological counseling 


14 


6 


Provide family support 


17 


10 


Provide career/college counseling 


36 


19 


Other support services activity 


4 


4 


Community outreach 


Provide outreach activities 


44 


31 


Establish partnerships 


48 


31 


Distribute mini- grants 


10 


8 


Other community outreach activity 


8 


5 



NOTE: Six. projects did not provide any information about the types of activities their projects were designed to conduct. ARC-funded activities 
represent those activities that respondents indicated were partially or fully funded by the ARC. 

SOURCE: 2000 mail survey of ARC grantees. 
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The remainder of this section explores the types of ARC-support efforts associated with each 
of the six categories of activities. Special emphasis is placed on the extent to which the data on Table 3-2 
reveal any noteworthy patterns by project characteristic. 

Developing Educational Resources. Almost four-fifths (78 percent) of projects reported an 
ARC-funded activity aimed at developing educational resources (Table 3-1). Not surprisingly, projects 
designed to benefit all elementary/secondary students were more likely than any other project type to 
conduct an educational resources activity (94 percent of projects compared with 60 percent for projects 
serving adults). In addition, projects serving two or more nonadjacent counties (96 percent) were more 
likely to cite this type of activity than were those serving two or more adjacent counties (77 percent) or 
those serving a single town or county (64 percent). Finally, almost all (91 percent) projects with only a 
single year of ARC funding were developing an educational resource compared with 82 percent of 
projects with 2 years of ARC funding and 63 percent of projects with 3 or more years of ARC funding. 

The two most widely conducted ARC-funded educational resource activities were providing 
teacher/tutor training (45 percent of projects) and developing teacher training programs/materials 
(32 percent) (Table 3-2). In addition, approximately one-quarter of projects used their ARC grant to 
develop paper-based educational materials (26 percent), develop computer-based educational materials 
(25 percent), and/or install other special use classrooms (24 percent). Only 5 percent reported using their 
ARC funds to install a science lab. Exhibit 3-1 provides examples of the educational resources that were 
developed with ARC support. 

Providing Training to Students and Other Community Residents. The second most 
frequently used category of activities involved the provision of training to students and other community 
residents (71 percent of projects) (Table 3-1). Most projects serving adults (92 percent) and targeted 
elementary/secondary students (86 percent) used their ARC funding to conduct at least one training 
activity. This is not surprising, given that the training activities most frequently cited focused on the 
provision of a workplace (e.g., computer) or remedial (e.g., literacy, GED preparation) skill that would 
typically be provided to a specially targeted population. 
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Exhibit 3-1, Examples of educational resource activities 



■ Michelin Learning Centers, The Centers’ curriculum for remediation involved an instructional 
module addressing each task on the Workplace Skills Inventory (WS1), so that individual results 
could be reviewed by instructors for areas needing improvement. In many cases, all that was 
required was a short coaching session on those tasks the WS1 indicated needed improvement. 
Although the project team researched existing adult basic skills software systems, they 
determined that no one system would be able to adequately meet the needs of all learners. They 
therefore decided that the learning centers should include an array of customized resources, such 
as selected basic skills computer software, books, workbooks, cassettes, and videos. Efforts were 
made to integrate job-related materials into the curriculum. A “prescription sheet” was also 
developed for instructors that matched skill weaknesses with appropriate lessons and a variety of 
resources to assist in remediation. 

■ Linking Individuals to Fundamental Training (LIFT), LIFT used a team approach that 
combined remedial education in fundamental skills with extensive case management. Through 
ARC funding, they developed curricula geared toward the ability level and learning style of the 
individual. 

■ Juniata Valley School District Curriculum Project, The Juniata Valley School District 
developed a model aquaculture site for a secondary school district. ARC funding assisted in the 
collection, review, and development of aquaculture curricula, models, and application of 
technology. In addition, a conference on aquaculture was held to promote the interchange of 
ideas, and materials were distributed to other educational institutions. 

■ Adult Skills and Academics Project (A.S,A,P,), The A.S.A.P., a cooperative venture of the 
West Virginia Department of Education and the Thanks a Million Foundation, was designed to 
address the basic skill needs of adults preparing to enter the workforce or adults presently 
employed at work sites in West Virginia. Through ARC funds, they coordinated the training of 
teachers in the delivery of the curriculum and instructional delivery and developed curriculum at 
the work site. 

■ Forward in the Fifth Math and Science Project, Forward in the Fifth provided two different 
hands-on workshops to train elementary school teachers: (1) to teach math teachers by concept, 
rather than through rote memorization, and (2) to teach science teachers to focus on teaching 
science throughout activity centers. 

■ Western Maryland Math-Science Teacher Training Project, This project was designed with 
two components. The first component was a summer/fall training session designed to equip 48 
teachers with skills to facilitate the integration of math and science at the elementary and middle 
school level. The second component of this project was to provide more indepth training for the 
teacher teams and access to consultants who would work with teachers within the counties to 
assist in the transition to an integrated curriculum in math and science. 

SOURCE: Document review of ARC education projects. 
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The two most widely covered training topics were academic skills (46 percent) and 
computers (45 percent) (Table 3-2). Other training activities included literacy (37 percent), job skills 
(35 percent), GED preparation (30 percent), parenting skills (23 percent), and peer tutoring (17 percent). 
Exhibit 3-2 provides examples of the training activities that were conducted with ARC support. 

Telecommunications Applications. Almost two-thirds (64 percent) of projects used their 
ARC funding to implement a telecommunications activity (Table 3-1). The use of ARC funds to conduct 
a telecommunications activity was most prominent among communitywide projects (78 percent), as well 
as those serving all elementary/secondary students (74 percent), and adults (72 percent). In contrast, only 
one (14 percent) of the seven projects serving targeted elementary/secondary students reported a 
telecommunications activity. In addition, projects serving a single town or county (88 percent) were more 
likely than those serving a wider geographic area to report using ARC funding for this purpose. Finally, 
projects serving at least one distressed county were less likely to report using their ARC grant for a 
telecommunications activity than were those serving no distressed counties (50 percent and 84 percent, 
respectively). 

Just over half (51 percent) of all projects reported using their ARC grant to install 
computers, while 26 percent reported installing or developing a computer network (Table 3-2). Exhibit 
3-3 provides examples of the telecommunications activities that were conducted with ARC support. 

Community Outreach. Over two-fifths (42 percent) of projects used their ARC funding to 
conduct a community outreach activity — e.g., establishing a partnership (31 percent) or providing 
outreach activities (31 percent) (Tables 3-1 and 3-2). Community outreach typically included serving as a 
center of activity (e.g., an information or funding clearinghouse) or facilitating a community activity. For 
example, Forward in the Fifth served as an information clearinghouse on best practices by providing 
mailings, a newsletter, and public relations assistance for local affiliates and the schools. The use of 
community outreach was more prominent among projects serving preschoolers (67 percent), targeted 
elementary/secondary students (57 percent), and communitywide projects (56 percent) — and less 
prominent among projects serving all elementary/secondary students (39 percent) or adults (32 percent). 
Exhibit 3-4 provides examples of the community outreach activities that were conducted with ARC 
support. 
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Exhibit 3-2. Examples of training activities 



■ Learn-By-Computer Literacy Program. Leam-By-Computer was designed to provide adult 
learners an opportunity to use computer-based instruction in basic skills and GED studies. The 
grantee coordinated and conducted training for 41 instructors on how to help adult learners in the 
use of computers for basic skills and on employability skills training. 

■ Basic Skills and Employability Development Project. This project involved students attending 
the Computer Center for intervention services. It was designed to enhance their basic skills 
development and academic/applied skills correlated to their vocational program. Intervention 
services, which included remediation in math, reading, English, and communication skills, were 
performed by vocational and academic instructors and instructional aides. These were designed to 
empower students to achieve competencies for the mandated Ohio Proficiency Test, obtain a high 
school diploma, and enter the competitive workforce or postsecondary education program. 

■ Technology Learning Center for Workforce Development. The center prepared economically 
disadvantaged residents for entry-level manufacturing jobs. Specifically, they trained 125-175 
participants over a 1-year period for such jobs including skills assessment, career counseling, 
introductory course work, training and practice, and pairing with a mentor company. 

■ Presbyterian Child Welfare Agency (PCWA). With financial assistance provided by ARC, 
PCWA proposed to open its second child care center in the Kentucky River Area Development 
District (KRADD) for infants and toddlers ranging from 2 weeks to 3 years of age. As part of the 
project, in-home and in-class instruction was provided to parents in order to improve parenting 
skills. Through the Northeast Mississippi Community Services, Inc., ARC funds were used as 
operational funds for the Mississippi Mobile Literacy Laboratory to provide basic skills and 
workplace literacy training to residents. Specifically, the laboratory provided a basic remediation 
computer course, a career match program, an Apple tutorial guide, and a career and social skills 
training program. 

■ Hindman Settlement House. Hindman provided a summer tutorial program for approximately 
20 dyslexic students from eastern Kentucky. These students were housed and trained in an 
intensive 6-week program in reading, mathematics, auditory perception, and social values. 

■ Smart Start. As part of the Smart Start project, the Parent Mentor Team developed and 
implemented a parent empowerment training program to enable parents to become effective 
participants in designing and choosing services for their children. 

■ Oak Ridge National Laboratory Summer Academy. The Summer Academy for High School 
Students and Teachers was conducted in collaboration with the Department of Energy. A summer 
research experience allowed students and teachers to participate in a 2-week hands-on science 
and math training programs with scientists at the Oak Ridge National Laboratory (ORNL) and 
Continuous Electronic Beam Accelerator Facility (CEBAF). During the last 2 years of funding, a 
Teacher Leadership Institute and a Student/Teacher Math/Science Institute were held for selected 
participants. The focus was on hands-on laboratory research used in ongoing projects. 

SOURCE: Document review of ARC education projects. 
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Exhibit 3-3. Examples of telecommunications applications 



■ Cherokee Adult Literacy Project. The Cherokee Adult Literacy Project was designed to expand 
the adult literacy services already provided. The ARC funding was used to purchase a computer 
that was used as an additional resource to the program by enabling students to work at the adult 
learning center at their convenience, rather than only when an instructor is available. The 
computers allowed for one-on-one instruction, which enabled the students to work at their own 
pace. 

■ Workplace Learning Program. The Workplace Learning Program for Dalton/Whitfield County 
installed 13 additional IBM computers as part of the learning demonstration lab to provide 
individualized instruction in reading, language, and math to increase the skill levels of employees. 

■ Learn-By- Computer Adult Literacy Program. Leam-By- Computer was designed to provide 
adult learners an opportunity to use computer-based instruction in basic skills and GED studies. 
Specifically, the project provided 15 computers at 15 different sites in the six-county area, in 
addition, printers and basic skills software were purchased to cover reading, math, English, and 
critical thinking skills. 

■ Write to Read. The Write to Read project worked in collaboration with IBM to establish 
programs in 10 comities. They coordinated and distributed hardware, software, and other 
equipment and materials purchased by IBM and supplied 10 printers for each site. 

■ After School Tutoring Program (ASTP). ASTP purchased computers, software, and additional 
equipment for the adult literacy lab and installed the center with a comprehensive computer 
software package for adult literacy. These centers were set up to test for literacy and scholarship 
programs, and support services were provided to attract individuals who would not normally be 
able to enter the program. 

■ Distance Learning Network Linkage. This network established new distance learning 
classrooms, which were expected to serve over 1,300 people during the first year of operation. 
The project connected 13 telecommunications classroom to the fiber optic backbone and linked 
the Erie I BOCES Project “CONNECT” interactive voice/video network. 

■ Visions Five Plus Telecommunications. Vision Five received ARC funding for the 
implementation and start-up of the Visions Five Plus interactive television network in public 
secondary schools in distressed comities. The overall purpose of the project was to establish a 
cooperative network of community learning centers. 

■ Diversified Technology Program. This program received ARC funding to purchase and install 
necessary equipment to establish a diversified technology lab for students to explore various uses 
of technology and jobs related to technology. 

SOURCE: Document review of ARC education projects. 
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Exhibit 3-4. Examples of community outreach activities 



■ Science Center of West Virginia. Special activities were held at and sponsored through the 
Science Center of West Virginia, such as four membership nights per year, which ranged in 
attendance from 150-300, depending upon the focus of the event. The focal point of these events 
ranged from new exhibits at the Center to guest speakers. The most recent membership night 
focused on a new computer designed to accommodate quadriplegic visitors. The members were 
invited to learn and test the new computer. As a special event for National Engineers Week, the 
Center sponsored an “egg-dropping contest” in which competing students from surrounding 
counties packaged eggs in unique ways in hope of having them survive a three-story drop from 
the top floor of the Science Center. 

■ Career Academy. The Career Academy was designed to prevent students in grades 6-8 from 
dropping out of school. The Career Academy represented a partnership between teachers, parents, 
students, business/industry, and other community members. The academy provided 60 potential 
dropouts and/or college-bound students, not achieving at maximum potential, with opportunities 
to link school work to real work and still maintain academic rigor. In addition, community 
outreach activities included making presentations to potential students and parents. 

■ North Carolina Rural Initiative-Outreach and Education Project. As part of the project, 
ARC provided funding to small towns and rural communities through mini-grants. Other 
activities included holding a series of regional meetings to explain and educate rural leaders about 
the rural initiative. 

■ Forward in the Fifth. Forward in the Fifth worked to encourage effective community and 
parental involvement that would help schools better meet the needs of young people. In order to 
achieve this goal, Forward in the Fifth served as an information clearinghouse on best practices, 
including providing mailings, a newsletter, and public relations assistance for local affiliates and 
the schools. 

■ Dropout Prevention/Career and Vocational Awareness Project. One of the major foci of the 
project was to form a solid partnership with the local business community to address their work 
skill needs through a vocation program designed jointly by business leaders and educators. 

■ Science and Math to Go! (SMTG!). Community support was critical to the launching of SMTG! 
In addition, volunteers contributed heavily to the refurbishment of kits. University student 
organizations, mostly from Clemson University, and retirees come on a regular basis to assist 
with unpacking used kits, preparing new materials to go in kits, and packing kits for distribution. 
Some retirees also help with locating additional corporate and community partners. A small 
number of retirees, Clemson faculty, and parents volunteer in classrooms helping teachers 
monitor students’ discovery activities. 

SOURCE: Document review of ARC education pro jects. 
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Providing Support Services. Over one-quarter (28 percent) of projects used their ARC 
funding to provide support services (Table 3-1). These services typically included case management or 
helping families find and integrate necessary services. For example, the Presbyterian Child Welfare 
Agency (PCWA) provided parents of participating children with case-managed assistance in enrolling in 
job readiness, academic upgrade, and vocational skills courses in coordination with other existing 
programs. Not surprisingly, five (71 percent) of the seven projects serving targeted elementary/secondary 
were providing support services. In addition, the high proportion of preschool projects that provided 
support services suggests that these initiatives sought to ameliorate a range of family-related issues. The 
most prominent service offered was career/college counseling (19 percent) (Table 3-2). Exhibit 3-5 
provides examples of the support services that were provided with ARC support. 

Improving Physical Plant. Just over one-quarter (27 percent) of projects reported at least 
one ARC-funded physical plant activity (Table 3-1) — most commonly installing or replacing mechanical 
equipment (17 percent) (Table 3-2). Exhibit 3-6 provides examples of the physical plant improvements 
that were conducted with ARC support. Almost one-third (32 percent) of elementary/secondary projects 
serving all students cited this approach, suggesting that ARC funds were being used to make 
improvements to school facilities. Additionally, projects serving only nondistressed counties (40 percent) 
were more likely to make such improvements than were those serving at least one economically 
distressed county (21 percent) or all counties in a state (6 percent). 
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Exhibit 3-5. Examples of support service activities 



■ School Outreach Project. School Outreach provided families of at-risk kindergarten and first 
grade students the individualized support necessary to maximize the likelihood that children 
arrive at school healthy and ready to learn. Most often, outreach workers helped children with 
reading and mathematics and other enrichment activities and educated families about hygiene, 
stressing the importance of bathing and watching for head lice. They also provided access to 
multiple resources, including referrals to mental health providers, assistance obtaining Medicaid, 
transportation to eye doctors and dentists, and assistance in navigating the public assistance and 
child care systems. 

■ Presbyterian Child Welfare Agency (PCWA). PCWA provided parents of participating 
children with case-managed assistance in enrolling in job readiness, academic upgrade, and 
vocational skills courses in coordination with other existing programs. 

■ The David School. The Success Bound program at the David School was a comprehensive 
approach to better prepare students who had dropped out of school to join the labor market. 
Students in the Success Bound program were provided with career exploration and preparation 
for life after graduation. Periodic meetings for students with a staff counselor were offered to 
explain the various services available and ensure that students’ needs are being met to the greatest 
extent possible. 

■ Linking Individuals to Fundamental Training (LIFT). LIFT delivered services on an in-home 
basis to 35 low-income adults who lacked the ability to engage in high school equivalency level 
training and were unable, due to transportation and/or other barriers, to access existing services or 
programs. Included were case management/psychological services. 

■ Drop-out Prevention/Career and Vocation Awareness and Training Project, This project 
worked to implement a worksite apprenticeship training program targeted toward at-risk high 
school youth who needed to increase their employment skills and work opportunities. The major 
focus of this project was to address the work skill needs of local business through a vocation 
program designed jointly by business leaders and educators. This project developed and 
implemented an apprenticeship and job-shadowing program for approximately 80 at-risk youth. 
In addition, it implemented a summer career exploration program to assist recent dropouts and at- 
risk students. 

■ Adult Basic and Literacy Education. This program of the Meigs County Schools included 
career awareness as part of their overall approach to educating adults. Specifically, learners were 
able to investigate career clusters, career skill information, and duties of workers, and to learn 
about resume and interviewing skills for jobs. 

SOURCE: Document review of ARC education projects. 
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Exhibit 3-6. Examples of efforts to improve physical plant 



■ Literacy Program to Provide Education Opportunities to Adults. The Literacy Program was 
designed to renovate an existing literacy center. The grant permitted the existing center to expand 
operation to a larger number of individuals by providing handicap accommodations and 
upgrading electrical services. 

■ Juniata Valley School District Curriculum Project. This project was designed to develop a 
model aquaculture site. Part of the ARC grant was used to design an appropriate tank or pond 
system to accommodate the needs of the project. 

■ Cullman County School District. The district received funding to provide temporary classrooms 
for students at two schools that were destroyed by fire. Fifteen classroom trailers and one office 
trailer were leased for 1 year, and 400 desks and office equipment were purchased for the 
classrooms and offices. 

■ McCreary County Learning Center Expansion Project. This project entailed equipping the 
learning center with furnishings to meet the educational, training, and retraining requirements of 
area residents. Risers and seating were provided for the lecture hall and the community 
conference room. In addition, the parking areas were renovated and expanded to accommodate 
the increased number of participants at the facility. Outdoor seating was also to be added to the 
campus. 

■ Itawamba County School District. The district purchased equipment to furnish a 
physics/chemistiy lab, biology lab, and a math computer lab, including furniture, and other 
necessary equipment in order to improve the math and science education in their school district. 

SOURCE: Document review of ARC education projects. 




3-13 



68 



3.1.2 



Combination of Activities Conducted by Projects 



Very few projects restricted their efforts to a single category of activities (10 percent) or 
attempted to implement an activity in all six categories (12 percent) (not shown in tables). In fact, the 
average educational initiative conducted activities in three different categories — suggesting that projects 
were fairly ambitious in the diversity of activities they chose to implement. 



A series of correlation analyses revealed that the projects in the study sample tended to 
implement particular combinations of activities. For example, projects that reported at least one physical 
plant activity also tended to report at least one telecommunications activity. Likewise, projects that 
reported a support service activity also tended to report a community outreach activity. Finally, projects 
reporting a support service activity also tended to report at least one training activity. Taken together, 
these patterns suggest that projects’ approaches tended to merge into a diversified, yet coherent strategy 
for achieving their goals. The following examples, taken from the case studies, illustrate this point: 

■ The Mobile Technology Project combined multiple activities, including 
the distribution of laptop computers, the installation and maintenance of 
a LAN using wireless infrared technology, the provision of training to 
teachers in how to use equipment and infuse technology into their daily 
lessons, the provision of instruction to students and their parents, and the 
establishment of community partnerships. (Mobile Technology Project) 

■ The Partnering with Parents for Successful Early Childhood 
Development project in eastern Tennessee developed written educational 
materials for parents (“Smart from the Start” sheets) and distributed them 
through local community agencies. It also provided classes in parenting 
skills and encouraged parents to complete their GED. (Partnering with 
Parents for Successful Early Childhood Development Project) 

■ The Science Center of West Virginia received a grant enabling the 
development of hands-on science exhibits. The center also developed 
science manipulative devices for classroom teachers to use, as well as a 
mobile, inflatable planetarium to use in classroom visits providing 
instruction in astronomy. (The Science Center of West Virginia) 



3.2 Extent of Implementation 



In an effort to assess the degree to which projects were able to implement their approaches, 
the mail survey obtained information on whether the level of achievement for each activity that a project 




3-14 



69 



conducted was “less than planned,” “same as planned,” or “more than planned.” 3 This section summarizes 
the extent of implementation across the 78 projects and 556 ARC-funded activities. All findings focus on 
those activities that were partially or fully funded by the ARC grant — i.e., no effort was made to analyze 
the extent of implementation for the activities that projects conducted without ARC support. 

An important caveat about these finding is worth noting. Data presented in this section 
represent descriptive comparisons of the extent of implementation of activities by project characteristics. 
These comparisons were often made across vastly different types of projects. For example, if a small 
project installed computers “more than planned” and a large project developed a distance learning 
network “less than planned,” we cannot infer that small projects were more successful than large projects. 
It is for this reason that we are hesitant to use these findings to make general policy recommendations 
about how the ARC might improve the management of its projects. (This issue is addressed in more detail 
in Chapter 6.) 

Over four-fifths (85 percent) of projects reported that they were able to implement their 
ARC-funded activities as planned 4 (Table 3-3). Interestingly, the likelihood of projects realizing their 
implementation goals was not related to any project characteristics. 5 

Only 1 5 percent indicated that their level of implementation on at least one task was less 
than planned, and no projects reported that they implemented all of their activities less than planned. 
Projects serving preschoolers (33 percent), targeted elementary/secondary students (29 percent), and 
adults (20 percent) were more likely than other project types to report that at least one of their ARC- 
funded activities was not fully implemented. In addition, statewide or multistate projects were more likely 



3 Although this approach does not allow us to know specifically what degree of implementation was planned, it does allow us to use a common 
metric across a broad range of activity types. Obviously, if a project had low expectations, it was easier to claim that a given activity was 
implemented more than planned. Conversely, a project with high expectations might be more likely to report that its level of accomplishment 
was less than planned. 

4 For the purposes of this discussion, “as planned” represents a combination of projects reporting on the survey that their level of implementation 
was same as planned or more than planned. 

5 Note that differences that appear large in magnitude may not be statistically significant at the 0.10 level. All findings reported in the text are 
statistically significant to the 0. 10 level. 
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BEST COPY AVAILABLE 



Table 3-3. Extent to which projects implemented reported activities, by project characteristics 



Project characteristic 


At least one activity less 
than planned 


All activities same as 
planned or more than 
planned 


All projects (n=78) 


15 


85 


Project type 


Preschool age (n=6) 


33 


67 


Elementary /secondary, all students (n=31) 


7 


94 


Elementary /secondary, targeted students (n=7) 


29 


71 


Adults (n=25) 


20 


80 


Communitywide (n=9) 


11 


90 


Project scope 


Single town or county (n=25) 


4 


96 


Adjacent counties (n=30) 


20 


80 


Nonadjacent counties (n=23) 


22 


78 


Economic status 


At least one distressed county (n=24) 


13 


88 


No distressed counties (n=38) 


13 


87 


Statewide or multistate (n=16) 


25 


75 


Metropolitan status 


Nonmetropolitan only (n=38) 


13 


87 


Metropolitan only (n=6) 


17 


83 


Both metro and nonmetro (n=18) 


11 


90 


Statewide or multistate (n=16) 


25 


75 


ARC grant size 


Less than $50,000 (n=l 7) 


12 


88 


$50,001 - $100,000 (n=22) 


5 


95 


$100,001 - $200,000 (n=25) 


20 


80 


More than $200,000 (n=14) 


29 


71 


Total project cost 


Less than $100,000 (n=18) 


6 


94 


$100,001 - $200,000 (n=27) 


19 


82 


$200,001 - $900,000 (n=25) 


16 


84 


More than $900,000 (n=8) 


25 


75 


Years of ARC funding 


1 year(n=21) 


14 


86 


2 years (n=33) 


12 


88 


3 or more years (n=24) 


21 


79 



NOTE: Six projects did not provide any information about the types of activities their projects were designed to conduct. No project reported that 
all of its activities were implemented less than planned. Percents may not sum to 100 due to rounding. 

SOURCE: 2000 mail survey of ARC grantees. 
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than other initiatives in these two categories to report that at least one of their ARC-funded activities was 
implemented less than planned (25 percent). However, there did not appear to be any differences in the 
implementation experiences of projects serving distressed and nondistressed counties. Finally, projects 
funded for 3 or more years were more likely than projects funded for 1 or 2 years to have implemented at 
least one ARC-funded activity less than planned (21 percent). 

These findings suggest that projects that were smaller in scope may have been better able to 
implement their approaches, while larger (and potentially more ambitious) projects were more likely to 
encounter at least some difficulty with at least one of their activities. While this is not a particularly 
surprising finding, it does ultimately suggest a set of criteria that ARC project coordinators might use in 
determining which types of projects are most likely to benefit from additional technical assistance and 
monitoring. It does not, however, provide a basis for suggesting a policy about which size of projects 
ARC should fund. As discussed previously, the study sample was too small to examine the influence of 
project size on a particular approach. 

In addition to determining whether projects were able to implement their planned activities, 
we also explored the extent of implementation across all of the activities. The primary purpose was to 
determine whether projects’ success was associated with the types of activities they tried to implement. 
Nearly all of the 556 ARC-funded activities were implemented as planned (Table 3-4). In fact, over one- 
fifth (21 percent) of activities were implemented more than planned. This pattern held across all 
categories of activities. 



3.3 Implementation Barriers 

Although the majority of ARC-funded activities were conducted as planned or more than 
planned, findings from both the mail survey and case studies suggest that projects did encounter some 
implementation barriers. In some cases, these obstacles were serious enough to prevent projects from 
fully implementing their ARC grant. In others, they merely hindered a project’s overall effectiveness or 
altered its implementation strategy — e.g., introducing time delays, limiting the range of services or 
potential beneficiaries. This section describes the barriers and problems that hindered projects’ efforts to 
conduct the activities that were partially or fully funded by the ARC. 
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Table 3-4. Extent to which projects implemented activities that were partially or fully funded by 
ARC, by the various activities that projects reported conducting 



Type of activity 




Extent of implementation 




Less than planned 


| Same as planned | More than planned 


Total (n=556) 


5 


74 


21 


Physical plant 


All physical plant activities (n=29) 


0 


79 


21 


Install/replace mechanical equipment (n=14) 


0 


79 


21 


Renovate structures (n=6) 


0 


67 


33 


Build new structures (n=6) 


0 


83 


17 


Other physical plant activity (n=3) 


0 


100 


0 


T elecommunic ations 


All telecommunications activities (n=82) 


4 


81 


16 


Install computers (n=43) 


2 


86 


12 


Install/develop network (n=22) 


0 


77 


23 


Develop distance education system (n=13) 


8 


69 


23 


Other telecommunications activity (n=4) 


25 


75 


0 


Educational resources 


All educational resources activities (n=150) 


4 


73 


23 


Install science lab (n=8) 


0 


88 


13 


Install other special use classroom (n=20) 


0 


95 


5 


Develop computer-based educational materials (n=21).... 


5 


81 


14 


Develop paper-based educational materials (n=22) 


0 


77 


23 


Develop teacher training program/materials (n=27) 


4 


67 


30 


Provide teacher/tutor training (n=38) 


8 


63 


29 


Other educational resources activity (n=14) 


7 


57 


36 


Training 


All training activities (n=200) 


5 


75 


21 


Provide literacy training (n=31) 


10 


71 


19 


Provide computer training (n=38) 


3 


76 


21 


Provide GED preparation training (n=25) 


0 


76 


24 


Provide job skills training (n=27) 


7 


76 


17 


Provide parenting skills training (n=19) 


11 


68 


21 


Provide academic skills training (n=39) 


0 


77 


23 


Provide peer tutoring (n=14) 


0 


79 


21 


Other training activity (n=5) 


20 


60 


20 


Support services 


All support services activities (n=32) 


6 


78 


16 


Provide emotional or psychological counseling (n=5) 


0 


100 


0 


Provide family support (n=8) 


0 


75 


25 


Provide career/college counseling (n=16) 


13 


75 


13 


Other support services activity (n=3) 


0 


67 


33 


Community outreach 


All community outreach activities (n=63) 


10 


64 


27 


Provide outreach activities (n=26) 


12 


65 


23 


Establish partnerships (n=26) 


8 


65 


27 


Distribute mini- grants (n=7) 


0 


71 


29 


Other community outreach activity (n=4) 


25 


25 


50 



NOTE: Percents may not sum to 100 due to rounding. 
SOURCE: 2000 mail survey of ARC grantees. 
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Survey respondents were provided a list of common implementation barriers and asked to 
indicate which ones pertained to their project efforts. One-third (34 percent) of projects reported that they 
had no implementation problems, while 21 percent reported only one barrier, 15 percent reported two 
barriers, 10 percent reported three barriers, and 14 percent reported four barriers (Figure 3-1). Five 
respondents (6 percent) reported five or more problems. 



As shown in Figure 3-2, no single obstacle was reported by more than one-third of the 
projects. The most frequently reported obstacle was underestimating the time or effort needed to 
implement the activities — with 34 percent of respondents indicating that their projects took more time or 
effort than originally anticipated. In addition, 19 percent of respondents indicated that participants had not 
maximized the use of their projects’ services, 15 percent cited local administrative delays, 15 percent 
indicated they had underestimated the demand for services, and 14 percent cited difficulties developing 
program materials. Projects also identified barriers not included as response options — e.g., the inadequacy 
of their facilities, inclement weather, and local political complications. 



The site visits provided an opportunity to delve more deeply into the types of problems that 
projects encountered. 6 As might be expected, given the unique nature of the activities these projects 
attempted, we uncovered some additional problems that were not listed as response options on the survey. 
Seven of the eight case study sites indicated that they encountered some resistance to their proposed 
approach. For example: 

■ In the School Outreach Project, K-12 teachers expressed concerns that 
project workers were undermining their authority. The outreach workers 
found that while most teachers were supportive of the School Outreach 
Project, a few were less receptive. Some teachers and principals new to 
the schools felt that the outreach workers were telling them that they 
were not doing their jobs right and that the outreach workers were there 
to watch over their teaching. Project staff felt these were front-end issues 
that were easily dealt with at the initial orientation session. 

■ In Adair County Technology Center project, school administrators 
indicated that a technology center was not a necessity in their county. 

Although parents and local businesses and industry quickly recognized 
the significance of the technology center, it was harder to sell its merits 
to the education community (e.g., faculty, administrators). The project 
director noted that it was necessary to persuade the superintendent of the 
center’s value, since his support trickled down to administrators and 
faculty and quieted their resistance. 



6 As discussed in Appendix E, these eight case study sites were not representative of the overall study sample — i.e., they were selected because 
they had implemented and sustained a unique approach that warranted further study. 
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Figure 3-1. Number of problems or barriers to implementing ARC-funded activities (n=80) 




Percent of projects 



SOURCE: 2000 mail survey of ARC grantees. 



The Michelin Learning Centers project experienced resistance on two 
fronts, one from the ranks of the production workers and one from 
management. After learning of the new company policy regarding basic 
skills testing, many of the production workers expressed concern and 
apprehension. Many feared for their jobs, and efforts were taken to 
reassure workers that although they would need to demonstrate sufficient 
basic skills in reading, math, and computer usage, they would be fully 
and patiently supported in the process. Some workers were reluctant to 
visit the learning centers, fearing the stigma and embarrassment attached 
to remediation. Further, many workers viewed voluntary continuing 
education as unnecessary and burdensome. Resistance to the learning 
centers also came from above, with some in management not buying in 
to the initiative. While the learning centers were fully implemented, two 
of the plants were not involved in project planning, and this exclusion 
hardened their resistance. According to the project’s final report, local 
plants were not involved early on and their input was sought only to 
determine how to implement the plan. The effect was a low level of 
involvement by some plant management. 
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Figure 3-2. Percent of projects reporting implementation barriers for ARC activities (n=80) 



Planning 




Percent of projects 

NOTE: Four projects did not provide any information about the implementation barriers their projects faced. 
SOURCE: 2000 mail survey of ARC grantees. 



Half of the case study sites faced budget constraints and/or had limited resources to 
implement and maintain their projects. In several cases, the respect that projects had earned throughout 
the community did not translate into financial support, in large measure because key stakeholders lacked 
the necessary resources. In another case, a project’s operational and maintenance costs were too high for a 
community that was in a period of economic decline (Exhibit 3-7). 
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Exhibit 3-7. Science Center of West Virginia 



The Science Center continually struggled to obtain the amount of money necessary to operate an 1 1,500 
square foot, high-maintenance facility that experiences a lot of traffic and heavy use of its exhibits by 
youth. Most of the surrounding area has experienced a population loss as a direct result of a serious, 
long-term decline in the economy. Some local industries have closed their doors, and others have been 
purchased by companies outside of the area. The founder/president of the Science Center commented, 
“The money we raised five and six years ago is no longer on the streets.” A few examples highlight this 
fact: the type of mom-and-pop stores (e.g., drug stores, hardware stores and book stores), that would 
have typically supported an enterprise such as the Science Center — for both civic reasons and because 
their extended families are integral parts of the schools — have been replaced by large retail chains 
located in the mall outside of town; the local wood pulp business is now owned by a company in 
Germany; the telephone company is owned by a firm in New Jersey; the electric company is owned by a 
company in a neighboring state; the newspaper and television station are owned out of state; and the 
locally owned Pepsi Cola bottling and distribution plant, which could always be relied upon to provide 
beverages at special events, has been sold. This situation severely limits the amount of money that is 
being recirculated back into the community for many purposes and has impacted not only on the 
Science Center’s larger fundraising efforts, but even the small donations it used to be able to obtain 
from local merchants in terms of services and supplies. 



Finally, in several case study sites, demand outpaced supply. This appeared to happen in 
cases where projects underestimated the staff or materials needed for a project — or the project 
experienced such difficulties as high staff turnover or lack of space to store project materials (see 
Exhibit 3-8). 



Exhibit 3-8. Science and Math To Go! 



Science and Math To Go! (SMTG!) had far more demand for professional development than it could 
handle. Teachers wanted to be trained on new kits, and new teachers wanted the initial training. The 
need to address new state standards and assessments strained the program’s training resources. 
Moreover, unlike science, there were no commercially available math kits for SMTG! to support. Staff 
and teachers had to create them and contract with a supplier for the materials. This was a long, slow 
process and drew time away from other activities, including providing professional development. One 
kit was created for each grade level, but demand for additional math kits continues. While everyone 
hoped that new kits would be developed in the coming year, researching and testing materials and then 
having kits assembled was a lengthy process. Storage space was also a problem. Teacher demand for 
kits overwhelmed the distribution capacity of the Materials Resource Center. While the center’s space 
had improved dramatically over the early years, it still needed a more efficient shelving system and 
space for cutting and measuring materials and repacking the crates. This will become even more of an 
issue in future years as SMTG! begins distributing kits for middle school and adding more math and 
additional science kits to meet the standards. 
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Taken together, the findings from the survey and case studies suggest that inadequate 
planning — especially estimating the time needed to implement a given task — represented the only 
obstacle that projects encountered to a large extent (and even this barrier was cited by only one-third of 
projects). The relative absence of grant-related barriers (e.g., grant not awarded in timely manner, 
insufficient funding) suggests that respondents generally viewed the level of support provided by ARC as 
being both timely and sufficient to meet their needs. 



3.4 Summary 

The projects in the study sample reported conducting a total of 682 distinct activities — 556 
of which were partially or fully funded by ARC. The majority of projects conducted an ARC-funded 
activity aimed at developing educational resources, providing training to students and other community 
residents, and/or installing/enhancing telecommunications applications. The average educational initiative 
conducted activities in three different categories — suggesting that projects were fairly ambitious in the 
diversity of activities they chose to implement. Across all activity types, the most frequently cited ARC- 
funded undertakings were installing computers, providing academic skills training, providing computer 
training, providing teacher or tutor training, and providing literacy training. 

Over four-fifths of respondents met or exceeded their implementation objectives. Only 15 
percent indicated that their level of implementation on at least one task was less than planned — and no 
projects reported that they implemented all of their activities less than planned. Nearly all of the 556 
ARC-funded activities were implemented as planned — in fact, 21 percent were implemented more than 
planned. Finally, the surveys and case studies provide little evidence that projects encountered any 
widespread barriers that affected the degree to which an activity was implemented. In fact, with the 
exception of underestimating the time/effort needed to conduct the project, no more than one-fifth of 
respondents cited any single barrier as having hindered their capacity to carry out the activities in their 
ARC grant. 
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4. ARC OUTCOMES 



The ultimate test of the merit of ARC-supported projects is whether they achieve their 
intended educational outcomes. Even initiatives that successfully implement all of their proposed 
activities cannot be considered successful until they provide evidence that they benefited the communities 
they serve. Such evidence might include gains in student achievement (e.g., increased grades, improved 
scores on statewide tests), increased interest in educational attainment (e.g., higher graduation rates, lower 
dropout rates, increased enrollment in higher level mathematics and science courses, increased enrollment 
in college), gains in adult learning (e.g., increased literacy rates, increased GED attainment rates), and 
gains in economic well-being (e.g., increased wages, decreased unemployment rates). 

This chapter assesses the extent to which the projects in the study sample achieved the 
educational objectives they delineated for themselves in their proposals to the ARC. 1 In reviewing these 
findings, it should be noted that many of the projects in the study sample were funded before the ARC 
revised its application procedures and began requiring that applicants delineate specific outputs and 
outcomes in their proposals. While some projects in the study sample identified a tangible outcome (e.g., 
"retain more students in the school district”), others described an abstract or impractical outcome (e.g., 
" break the generational illiteracy cycle in the four county region ”). Few projects provided a numeric 
benchmark against which their progress could ultimately be measured. Had these projects been funded 
after 1 998 — the year in which the ARC began requiring that applicants specify measurable outcomes — it 
is likely that a higher proportion of the applications would have contained tangible and measurable 
outcomes. 



It is also important to note that we were not able to substantiate projects’ claims that they 
had achieved their ARC-supported outcomes. Although we asked projects to provide written evidence 
that a given outcome had been achieved, we were not in a position to assess whether these gains had, in 
fact, occurred. Nor were we able to ascertain whether the methodologies they used to document these 
gains were appropriate. 

Finally, with most grantees receiving funding and support from multiple sources, we were 
not able to determine the extent to which a project’s accomplishments were directly or indirectly 



1 See Appendix B for a discussion of projectwide accomplishments — i.e., accomplishments associated with the sum of activities conducted by 
projects (regardless of funding source). 
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attributable to the ARC grant. However, this chapter does discuss whether respondents believed they 
would have been able to achieve their outcomes if ARC funding had not been made available. 



4.1 Methodology 

In an effort to assess the extent to which grant recipients achieved the goals they had set for 
themselves, we reviewed each project’s application materials to identify die specific objectives and 
outcomes they had delineated in their original proposals to the ARC. These outcome statements were 
entered into a database that was then used to generate an addendum to the mail survey. This addendum, 
customized for each project, provided respondents with an opportunity to (1) indicate the extent to which 
they met their own outcomes, (2) provide tangible evidence that a given outcome was achieved, and (3) 
describe factors that hindered their ability to achieve a desired goal. 

To the extent possible, the phrasing for the outcomes provided in the addendum was taken 
from projects’ original applications to the ARC. In a limited number of cases, however, the outcome 
statements either had to be imputed (e.g., from more general language in the proposal) or taken from other 
sources (e.g., the official contract between the ARC and the grant recipient). Once they received the 
survey, respondents could contact Westat if they believed that the outcomes listed in the addendum were 
not representative of what their ARC project was designed to achieve. (None did so.) Respondents could 
also provide information about any additional ARC-related outcomes that were not already listed in the 
addendum. (Nineteen projects described additional outcomes.) 

Twelve (14 percent) of the 84 projects identified intended outputs (e.g., provide training to 
20 teachers ) — but no intended outcomes — in their ARC proposal. These 12 projects were excluded from 
the analysis of ARC-related outcomes. (We do, however, provide a separate discussion of the extent to 
which these 12 projects achieved their intended outputs in Section 4.3.) Some projects listed a 
combination of outcomes and outputs in their proposals. In such cases, we focused exclusively on 
outcomes and did not assess the extent to which projects’ intended outputs were achieved. 2 



” The decision to exclude outputs in cases where outcomes were also listed was based on two factors. First, the primary focus of the Government 
Performance and Results Act (GPRA) is on results and outcomes — as opposed to activities and outputs. Second, Item 7 of the mail survey was 
designed to obtain information on whether projects achieved their anticipated ARC-supported activities. In an effort to alleviate response 
burden, we concluded that there was no compelling reason for using the addendum to collect similar data about outputs (except in cases where 
projects failed to identify any outcomes). 





The primary intent of this exercise was to provide ARC with an assessment of the extent to 
which the education projects in the study sample produced tangible gains — e.g., an increase in student 
achievement, adult literacy, or wages and job retention. A secondary purpose was to assess whether 
projects had more or less difficulty achieving specific types of outcomes — e.g., were projects designed to 
increase student achievement more successful than those designed to boost household income. As is 
discussed in Chapter 6, a third purpose was to develop an inventory of outcomes that could serve as a 
framework for enhancing ARC’S capacity to monitor and track its educational projects. 



4.2 Outcomes Associated with ARC-Funded Activities 

A major purpose of this evaluation was to evaluate projects “according to the performance 
standards set forth in the project description of objectives” {Request for Proposals by the Appalachian 
Regional Commission for A Program Evaluation of ARC’s Educational Projects, May 28, 1999). Unlike 
the projectwide activities described in Chapter 3, this section focuses on the specific goals that projects 
described in their original requests for ARC support — with an emphasis on the outcomes that were 
directly associated with their ARC grant. 



4.2.1 Types of ARC Outcomes That Projects Identified for Themselves 

Across Projects. Over four-fifths (86 percent) of projects in the study sample described at 
least one anticipated outcome or community benefit in their proposal to the ARC. These outcome 
statements generally outlined a skill (e.g., increase the reading and math skills of those participating in 
the program) or circumstance (e.g., retain more students in the school district) that would he enhanced as 
a result of their ARC grant. However, only a few of the outcomes described in the application materials 
had a measurable benchmark against which project success could be measured (e.g., reduce the dropout 
rate in Madison County by 50 percent in the next ten years, 20 percent or 7 individuals will upgrade their 
fundamental skill levels and social skill capabilities). Most projects used more general — and less 
quantifiable — language to describe their desired goals (e.g., boost the number of GED graduates, improve 
job performance of adult learners). In addition, a small number of projects outlined a lofty goal that was 
beyond the possibility of a single project to achieve (e.g., break the generational illiteracy cycle in the 
four county region, improve the overall quality of life for the citizens of SDA One). 
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Projects anticipated an average of 2.8 ARC-related outcomes — with projects serving 
preschoolers and targeted elementary/secondary students anticipating an average of 3.6 and 3.9 outcomes, 
respectively (Table 4-1). Not surprisingly, over four-fifths (83 percent) of projects described in their 
proposal an educational attainment objective that would occur as a result of the ARC grant (Table 4-2). in 
addition, 36 percent cited an economic benefit, and another 36 percent outlined a family/individual well- 
being outcome. An analysis of anticipated outcomes by project characteristics uncovered some interesting 
trends. Specifically: 



■ Project type. Most projects serving preschoolers and all projects serving targeted 
elementary/secondary students anticipated both educational and family/individual 
well-being outcomes — suggesting that these projects were seeking to ameliorate both 
educational and social barriers. In addition, the majority of projects serving adults 
anticipated educational gains (70 percent), and less than half anticipated economic 
(44 percent) or family/individual (26 percent) outcomes. 

■ Project scope. Nearly all projects situated in a single town or county (96 percent) 
anticipated an educational outcome, and approximately one-quarter (23 percent) 
described a family/individual well-being outcome. This suggests that local initiatives 
primarily focused on improving the educational status of a defined population (e.g., 
students in a single school or district), while regional initiatives were more likely to 
address economic concerns. 

Across Outcomes. Of the 203 outcomes identified during the document review, 122 
(60 percent) described an educational benefit, 42 (21 percent) described an economic benefit, and 39 
(19 percent) described a family/individual well-being outcome (Table 4-3). Exhibit 4-1 provides 
examples of projects’ anticipated educational, economic, and family/individual well-being outcomes. 
(Exhibits A-l (educational), A-2 (economic), and A-3 (family/individual well-being) in Appendix A 
provide additional examples of the outcomes that projects described in their proposals to the ARC.) An 
analysis of these outcomes uncovered the following noteworthy and statistically significant trends 
(Table 4-3): 



Project type. Not surprisingly, projects serving youth tended to identify education- 
oriented objectives. In addition, half (50 percent) of the 74 outcomes identified by 
projects serving adults described an educational benefit, 24 (32 percent) described an 
economic benefit, and 13 (18 percent) described a family/individual well-being 
benefit. 

Project scope. Over three-quarters (58 projects — or 77 percent) of the 75 outcomes 
identified by projects serving a single town or county described an educational 
benefit, as did 34 projects (47 percent) serving two or more adjacent counties and 30 
projects (55 percent) serving two or more nonadjacent counties. 
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Table 4-1. Average number of ARC outcomes reported by projects, by project characteristics 

Project characteristic | Average number of ARC outcomes 

All projects (n=72) 2.8 

Project type 

Preschool age (n=5) 

Elementary/secondary (all students) (n=25) 

Elementary/secondary (targeted students) (n=8) 

Adults (n=27) 

Communitywide (n=7) 



Project scope 

Single town or county (n=26) 2.9 

Adjacent counties (n=25) 2.9 

Nonadjacent counties (n=21) 2.6 



3.6 

2.6 
3.9 
2.7 
2.3 



Economic status 

At least one distressed county (n=24) 2.8 

No distressed counties (n=37) 2.9 

Statewide or multistate (n=l 1) 2.5 



Metropolitan status 

Nonmetropolitan only (n=37) 3.0 

Metropolitan only (n=7) 2.6 

Both metro and nonmetro (n=l 7) 2.7 

Statewide or multistate (n=l 1) 2.5 



ARC grant size 

Less than $50,000 (n=l 5) 2.3 

$50,001 - $100,000 (n=23) 2.9 

$100,001 - $200,000 (n=24) 2.9 

More than $200,000 (n=10) 3.2 



Total project cost 

Less than $100,000 (n=l 8) 2.6 

$100,001 - $200,000 (n=25) 2.8 

$200,001 - $900,000 (n=23) 3.0 

More than $900,000 (n=6) 2.7 

Years of ARC funding 

1 year(n=18) 2.7 

2 years (n=28) 3.0 

3 or more years (n=26) 2.1 



NOTE: Projects that only listed outputs are not included on this table. 
SOURCE: 2000 mail survey of ARC grantees. 
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Table 4-2. Percent of projects reporting at least one ARC outcome in each type of outcome, by 
project characteristics 



Project characteristic 


Increase educational 
attainment/preparation 


Increase economic 
well-being 


Increase family 
or individual 
well-being 


All projects (n=72) 


83 


36 


36 


Project type 


Preschool age (n=5) 


80 


40 


80 


Elementary/secondary (all students) (n=25) 


92 


24 


20 


Elementary/secondary (targeted students) (n=8) 


100 


38 


100 


Adults (n=27) 


70 


44 


26 


Communitywide (n=7) 


84 


43 


29 


Project scope 


Single town or county (n=26) 


96 


27 


23 


Adjacent counties (n=25) 


72 


48 


44 


Nonadjacent counties (n=2l) 


81 


33 


43 


Economic status 


At least one distressed county (n=24) 


96 


38 


42 


No distressed counties (n=37) 


81 


38 


32 


Statewide or multistate (n=l 1) 


64 


27 


36 


Metropolitan status 


Nonmetropolitan only (n=37) 


92 


41 


35 


Metropolitan only (n=7) 


71 


29 


29 


Both metro and nonmetro (n=17) 


82 


35 


41 


Statewide or multistate (n=l 1) 


64 


27 


36 


ARC grant size 


Less than $50,000 (n=15) 


80 


40 


53 


$50,001 -$100,000 (n=23) 


87 


22 


26 


$100,001 -$200,000 (n=24) 


83 


42 


42 


More than $200,000 (n=10) 


80 


50 


20 


Total project cost 


Less than $100,000 (n=l8) 


89 


33 


44 


$100,001 -$200,000 (n=25) 


80 


24 


40 


$200,001 - $900,000 (n=23) 


87 


48 


30 


More than $900,000 (n=6) 


67 


50 


17 


Years of ARC funding 


1 year (n=18) 


94 


33 


44 


2 years (n=28) 


79 


43 


29 


3 or more years (n=26) 


81 


31 


39 



NOTE: Projects that only listed outputs are not included on this table. 
SOURCE: 2000 mail survey of ARC grantees. 
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Table 4-3. Number of anticipated outcomes for each type of outcome, by project characteristics 



Project characteristic 


Increase 

educational 

attainment/ 

preparation 


Increase 

economic 

well-being 


Increase 
family or 
individual 
well-being 


Total 


Total 


122 


42 


39 


203 


Project type 


Preschool age 


10 


3 


5 


18 


Elementary/secondary (all students) 


50 


8 


6 


64 


Elementary/secondary (targeted students) 


15 


3 


13 


31 


Adults 


37 


24 


13 


74 


Communitywide 


10 


4 


2 


16 


Project scope 


Single town or county 


58 


9 


8 


75 


Adjacent counties 


34 


21 


18 


73 


Nonadjacent counties 


30 


12 


13 


55 


Economic status 


At least one distressed county 


44 


12 


12 


68 


No distressed counties 


63 


23 


22 


108 


Statewide or multistate 


15 


7 


5 


27 


Metropolitan status 


Nonmetropolitan only 


76 


19 


18 


113 


Metropolitan only 


11 


2 


5 


18 


Both metro and nonmetro 


20 


14 


11 


45 


Statewide or multistate 


15 


7 


5 


27 


ARC grant size 


Less than $50,000 


16 


7 


12 


35 


$50,001 -$100,000 


48 


8 


10 


66 


$100,001 -$200,000 


39 


17 


14 


70 


More than $200,000 


19 


10 


3 


32 


Total project cost 


Less than $100,000 


30 


7 


10 


47 


$100,001 -$200,000 


41 


10 


19 


70 


$200,001 - $900,000 


44 


18 


8 


70 


More than $900,000 


7 


7 


2 


16 


Years of ARC funding 


1 year 


30 


9 


9 


48 


2 years 


52 


20 


12 


84 


3 or more years 


40 


13 


18 


71 



SOURCE: 2000 mail survey of ARC grantees. 
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Exhibit 4-1. Examples of outcomes that projects 
specified in their original proposal to the ARC 



Educational Outcomes 

Increase adult literacy rates within the State of West Virginia. 

Break the generational illiteracy cycle in the four county region. 

Reduce the number of at-risk students in the 10 counties in the Appalachian region of Tennessee. 

Reduce the drop-out rate in the 10 counties in the Appalachian region of Tennessee. 

Retain more students in the school district. 

Increase the number of students who pursue studies in math and/or science. 

Economic Outcomes 

Attract new business and industry to the area. 

Upgrade adults 9 re-employment skills. 

Reduce the likelihood of plant closures and layoffs in the 6 participating counties. 

5 students who participated in the Success Bound program will gain employment. 

Reduce the unemployment rate in the 5 county region. 

Create additional non-farm income. 

Familv/individual Well-Being Outcomes 

Enable adult learners to be less economically dependent on the social service system. 

Improve worker self-esteem and job attitude. 

Return men and women to the community as productive individuals. 

Seventy-five percent or 26 individuals served will experience a significant increase in the quality of their 
lives, and a significant decrease in dejection and vulnerability to exploitation. 

Create a positive home climate that enhances positive feeling, relationships and aspirations , and connect 
home to the school and mainstream community. 

SOURCE: Document review of ARC education projects. 
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Economic status. Of the 68 outcomes identified by projects serving at least one 
distressed county, 44 (65 percent) described an educational benefit, 12(18 percent) an 
economic benefit, and 12 (18 percent) a family/individual well-being benefit. 

Metropolitan status. Of the 113 outcomes identified by nonmetropolitan projects, 76 
(67 percent) described an educational benefit, 19 (17 percent) an economic benefit, 
and 18 (16 percent) a family/individual well-being benefit 

Total project cost. Larger projects (i.e., those with total budgets of more than 
$200,000) were more likely than smaller projects to describe an economic benefit — 
suggesting that these projects had the resources to tackle issues related to the local 
economy. 

Years of ARC funding. Projects that received 3 or more years of ARC funding were 
more likely to describe a family/individual well-being outcome than those receiving 
only 1 to 2 years of ARC funding. 



4.2.2 Extent to Which Projects Achieved Their ARC Outcomes 



Almost three-quarters (74 percent) of respondents reported that they met or exceeded their 
anticipated level of achievement for all of their ARC outcomes — with 14 percent indicating that they had 
achieved all of their outcomes more than originally expected (Figure 4-1). Only 22 percent reported that 
their level of achievement on at least one outcome was less than expected — with only 7 percent indicating 
that their level of achievement was less than anticipated for all of their outcomes. In addition, projects’ 
level of achievement across different outcome types was similar — although respondents were most likely 
to report having achieved all of their family/individual well-being outcomes. Specifically; 



Educational attainment outcomes. Of the 60 projects that described an educational 
outcome, 77 percent indicated that they met or exceeded all of their anticipated 
educational goals — with 20 percent reporting that they achieved all of their 
educational outcomes more than planned (Figure 4-2). Only one-fifth (1 9 percent) had 
at least one educational outcome that was achieved less than planned. 

Economic well-being outcomes. Of the 26 projects that described an economic 
outcome, 81 percent indicated that they met or exceeded their anticipated economic 
goals (Figure 4-3). In fact, 15 percent achieved all of their economic outcomes more 
than planned. However, 19 percent indicated that their level of achievement across all 
of their economic outcomes was less than planned. 

Family/individual well-being outcomes. Of the 26 projects that described a 
family/individual well-being outcome, 78 percent indicated that their level of 
achievement was similar to or better then originally anticipated — and 31 percent 
achieved all of their family/individual well-being outcomes more than planned 
(Figure 4-4). Only 8 percent indicated that they achieved a family/individual well- 
being outcome less than planned. 



O 
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Figure 4-1. Percent of projects’ ARC outcomes, by the extent in which they were achieved (n=72) 




■ All achieved less than expected 

® At least 1 achieved less than 
expected and at least 1 achieved 
same as or more than expected 

O All achieved same as expected 

® All achieved same as expected or 
more than expected 

E3 All achieved more than expected 

^ Level of achievement not reported 



NOTE: Outcomes for which the extent of achievement was not reported are not included here. Percents may not sum to 100 due to rounding. 
SOURCE: 2000 mail survey of ARC grantees. 



Figure 4-2. Percent of projects that anticipated outcomes that were designed to increase 
educational attainment, by the extent to which they were achieved (n=60) 



20 % 



17% 




® All achieved less than expected 

® At least 1 achieved less than 
expected and at least 1 achieved 
same as or more than expected 

O All achieved same as expected 

® All achieved same as expected or 
more than expected 

^ All achieved more than expected 

^ Level of achievement not reported 



NOTE: Outcomes for which the extent of achievement was not reported are not included here. Percents may not sum to 100 due to rounding. 
SOURCE: 2000 mail survey of ARC grantees. 
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Figure 4-3. Percent of projects that anticipated outcomes that were designed to increase economic 
well-being, by the extent to which they were achieved (n=26) 



15% 



50% 




■ All achieved less than expected 

□ At least 1 achieved less than 
expected and at least 1 achieved 
same as or more than expected 

E3A11 achieved same as expected 

^ All achieved same as expected or 
more than expected 



NOTE: Outcomes for which the extent of achievement was not reported are not included here. Percents may not sum to 100 due to rounding. 
SOURCE: 2000 mail survey of ARC grantees. 



Figure 4-4. Percent of projects that anticipated outcomes that were designed to increase family or 
individual well-being, by the extent to which they were achieved (n=26) 




12 % 



■ All achieved less than expected 

^ At least 1 achieved less than 
expected and at least 1 achieved 
same as or more than expected 

□ All achieved same as expected 

^ All achieved same as expected or 
more than expected 

^ All achieved more than expected 

^ Level of achievement not reported 



NOTE: Outcomes for which the extent of achievement was not reported are not included here. Percents may not sum to 100 due to rounding. 
SOURCE: 2000 mail survey of ARC grantees. 
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4.2.3 



Extent to Which Specific Types of Outcomes Were Achieved 



In addition to assessing whether projects were able to achieve the sum of their anticipated 
outcomes, we also examined the extent of achievement across the three outcome types — i.e., educational, 
economic, and family/individual well-being. A primary purpose was to determine whether projects’ self- 
reported levels of success were associated with the types of outcomes they were trying to achieve. A 
secondary purpose was to develop an inventory of the evidence, by outcome type, that projects compiled 
to assess their own level of achievement. (Respondents were asked to provide quantitative or qualitative 
evidence that the level of achievement for any given outcome matched or exceeded their expectations.) Of 
particular interest was whether projects employed valid and credible assessment methods — and whether 
those methods could be easily adapted by other sites. The remainder of this section discusses what was 
found, and Chapter 6 discusses the implications of these findings for future data collection activities. 

Just under half (47 percent) of the 203 outcomes that projects identified in their ARC 
proposals were ultimately achieved as planned, while 30 percent were achieved more than planned and 1 3 
percent were achieved less than planned (Table 4-4). 3 The level of achievement was generally highest for 
family/individual well-being outcomes and lowest for economic outcomes. 



Table 4-4. Number and percent of ARC outcomes, by the extent to which they were achieved 



Type of ARC outcome 


Number of 
anticipated 
outcomes 


Extent of achievement (percent) 


Less then 
planned 


Same as 
planned 


More than 
planned 


Not 

reported 


Increase educational attainment 


122 


12 


47 


30 


12 


Increase economic well-being 


42 


24 


52 


24 


0 


Increase family/individual well-being 


39 


8 


44 


36 


13 


Total 


203 


13 


47 


30 


9 



NOTE: Percents may not sum to 100 due to rounding. 
SOURCE: 2000 mail survey of ARC grantees. 



There was considerable variety in the quality of the evidence provided in support of these 
outcomes — with no apparent correlation between levels of achievement and the credibility of the methods 
used to document projects’ accomplishments. It should be noted that we were not in a position to assess 
whether any of the evidence that projects described was credible and reliable. Nor were we in a position 



3 Respondents were not able to assess their level of achievement for the remaining 9 percent of outcomes. In most instances, this occurred 
because a given respondent had not been with a project at the time the proposal was written and was therefore unwilling to comment on the 
extent to which the outcome had been accomplished as planned. 
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to assess whether a specific gain (i.e., improved test scores, decrease in unemployment) was actually a 
result of the activities that ARC supported. As such, our inclusion of a project’s evidence should not be 
viewed as confirmation that the corresponding outcome was, in fact, achieved — or that the methodology 
was suitable and/or applied in an appropriate manner. 

Overall, the documentation provided for 83 (53 percent) of the 157 successfully achieved 
outcomes was derived from a specific data source (e.g., scores on state or national tests, decrease in 
dropout rates, job placement for project participants) or study (e.g., parent/teacher survey, pre/post survey 
regarding job skills) (not shown in tables). Much of the remaining documentation merely described a new 
or expanded service (26 percent of successfully achieved outcomes) or represented anecdotal evidence 
(14 percent). Claims of success that are derived from a specific study generally carry more weight than 
those based on activities or anecdotes. For example, one project that anticipated enhancing students’ 
computer skills provided the following as evidence: “Number of students served— without this grant the 
services would have been unavailable. " While it is possible that students’ computer skills were, in fact, 
enhanced, the evidence does not support this claim. A stronger form of evidence would have been a 
pre/post test of students’ computer skills and frequency of use. 



Educational Outcomes. Of the 122 educational outcomes that projects delineated in their 
proposals, 57 (47 percent) were achieved the same as planned — and 37 (30 percent) were achieved more 
than planned (not shown in tables). Only 14 (12 percent) of the 122 educational outcomes that projects 
delineated in their proposals were achieved less than planned. 



The evidence provided for 46 (49 percent) of the 94 educational outcomes that were 
achieved was derived from a specific study or data source (not shown in tables). These included: 

■ Increases in students’ scores on state or national tests — cited as the primary source of 
documentation for 21 percent of the 94 educational outcomes that were achieved as 
expected (Table 4-5). For example, one project with a goal of attaining “higher test 
scores in various testing situations" reported that “For the first time in school history , 
ACT scores for Wayne County High School students matched the state average . " 

■ Increases in the number of adults attaining a GED (11 percent). While some projects 
provided imprecise evidence (e.g., “The number of students that participated in this 
project and the number that received their GED was more than expected"), others 
provided more explicit documentation (e.g., “High school diploma— 30; GED— 272; 
this is what was expected"). 
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Table 4-5. Forms of evidence that projects used to support their claims that they met or exceeded 
a given outcome, by level of achievement 





Level of achievement (percent) 


Form of evidence provided in mail survey 


Same 

as expected 


More 

than expected 


Same/more 
than expected 


Education Outcomes (n=94) 


Description of new or expanded service 


21 


38 


28 


Scores on state or national tests* 


25 


16 


21 


Anecdotal description (e.g., gain in student interest) 


16 


16 


16 


Increase in adults attaining a GED* 


9 


14 


10 


Decreased dropout rate/increased school attendance or enrollment* 


11 


5 


9 


Parent/teacher survey* 


2 


5 


3 


Other (e.g., census data, postsecondary plans)* 


7 


3 


5 


Unclear (i.e., could not discern from survey response) 


11 


3 


7 


Economic Outcomes (n=32) 


Description of new or expanded service 


23 


3 


25 


Job placement for project participants* 


9 


3 


16 


Jobs created/businesses located in region* 


9 


3 


16 


Increase in job skills* 


18 


0 


13 


Ajiecdotal description (e.g., discussions with employers) 


14 


1 


13 


Increase in employment rate/decrease in unemployment rate* 


9 


0 


6 


Increase in degree attainment (e.g., from vocational program) * 


9 


0 


6 


Increase in wages* 


5 


0 


3 


Increase in productivity* 


5 


0 


3 


Family/Individual Well-Being Outcomes (n=31) 


Pre/post survey/observations regarding skills/interests/status* 


18 


36 


26 


Description of new or expanded service 


18 


21 


19 


Decrease in reliance on government assistance* 


18 


7 


13 


Job placement* 


18 


7 


13 


Anecdotal description (e.g., discussions with teachers) 


6 


14 


10 


Increase in adults attaining a GED* 


6 


0 


3 


Other/none 


18 


14 


16 



NOTE: Because this table uses outcomes — as opposed to projects — as the unit of analysis, the number of observations exceeds 84. In addition, 
this table does not make any adjustments for projects that used the same form of evidence to demonstrate achievement for multiple outcomes. For 
example, evidence submitted twice by the same project (e.g., number of jobs created) in support of two distinct goals would be counted twice on 
the table. Finally, for the purpose of this analysis, foims of evidence that are followed by an (*) are considered to be “a specific study or data 
source.” 

SOURCE: 2000 mail survey of ARC grantees. 
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BEST COPY AVAILABLE 



■ Decreases in school dropout rates/increases in school attendance/enrollment 
(9 percent). For example, a project with a goal of making " 'math and science a priority 
for Falkner High School Students " reported “More students are enrolling in advanced 
math and science courses. ” 

Fifteen (16 percent) of the 94 descriptions of project success were based on anecdotal 
accounts (e.g., teachers’ assertions that students are more interested in their schoolwork), while 26 
(28 percent) of the narratives merely recounted a new or enhanced service that was being made available 
to a given population (e.g., “Humber of students served — without this grant the services would have been 
unavailable”). 4 Exhibit A-4 in Appendix A provides a sample of the narratives that projects provided in 
support of the educational outcomes that were successfully achieved. 

Fourteen (11 percent) of the 122 educational outcomes that projects delineated in their 
proposals were achieved less than planned. As shown in Exhibit 4-2, no single barrier was responsible for 
these shortfalls. Rather, a variety of factors hindered projects’ efforts to achieve their desired results, 
including serving fewer participants than originally anticipated, setting goals that ultimately exceeded the 
project’s capacity, encountering factors beyond the control of the project (e.g., high unemployment rates), 
difficulty quantifying project results, and failing to fully implement the project. 

Economic Outcomes. Of the 42 economic outcomes anticipated by projects, 52 percent 
were achieved the same as planned — and 24 percent were achieved more than planned. However, almost 
one-fourth (24 percent) of all economic outcomes were achieved less than planned — suggesting that 
projects described overly ambitious economic goals and/or the economic benefits that projects anticipated 
were beyond the control of their ARC grant. 



The evidence provided for 20 (63 percent) of the 32 successfully achieved economic 
outcomes was derived from a specific study or data source. These included: 

■ Job placement for project participants (cited as the primary source of documentation 
for 16 percent of the 32 economic outcomes that were successfully achieved). For 
example, one project with a goal of making individual students more competitive in 
the job market reported, “In the two years since the completion of the project, several 
students have obtained jobs in computer operation. This evidence is obtained in a 
post-school survey done by the school. " 



4 An analysis of evidence between outcomes achieved same as planned and those achieved more than planned uncovered no noteworthy patterns. 
In fact, a higher proportion of the outcomes achieved more than planned were linked to a description of a new/expanded service than were 
outcomes achieved the same as planned. 
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Exhibit 4-2. Examples of factors that respondents indicated hindered 
their projects’ ability to achieve desired educational outcomes 



Anticipated outcome 


Factor that hindered level of achievement 


Increase computer literacy for adults in Meigs County 


Fewer participants were involved compared to the anticipated 
number we had hoped to reach. Many dropped out of the 
program before results could be measured. 


Increase basic skill levels in reading, math, and writing for 
adults in Meigs County 


We documented the increased skill levels. However, we 
reached fewer participants than anticipated. 


Increase the numbers of adult learned in the quad-county area 


Information was made available, but results are impossible to 
quantify. 


Break the generational illiteracy cycle in the four-county 
region 


The scope was too broad for the project. 


Prepare adults with the necessary basic skill remediation or 
GED preparation to gain entry level employment 


The high unemployment rates at that time affected the ability 
of students to find local employment. 


Enhance the educational level of the workforce in Madison 
County 


Actually, I don’t know! I’m not aware of how much was 
expected on any of these questions. 


Increase the number of students participating in science fairs in 
the Tuscaloosa areas, as well as in surrounding counties 


In 1992, science fairs and projects seemed to be the core of 
elementary science curricula. Over time, this has abated. The 
center has been used minimally for science fair projects. 


Move unemployed and underemployed families with 
marketable skills into long-term job training 


We served only 10 toddlers instead of the 20 reported in the 
grant — due to not having new building finished — so we were 
able to help less parents than anticipated. 


Increase in high school graduation rates for children who 
participate in the program 


We really cannot address this until these children reach high 
school age. 


Decrease dropout rate by offering a wider variety of 
stimulating, hands-on learning courses and vocational Tech- 
ED courses 


The dropout rate has increased slightly. This is partly attributed 
to the importance of school not being taught at home and 
partly associated with increased graduation requirements. 


Improve the educational needs of the five distressed counties 


The network was placed under the control of the schools. 
Teachers were afraid of the technology. Many feared loss of 
their jobs, unjustly, if the project took hold, so they sought to 
subvert the system and did so. The secondary schools resented 
overtures from community colleges to offer courses, fearing 
that would lead to effective use of the system. So 
postsecondary and vocational schools had the network 
removed for fear instructors would be displaced and the 
authority of the director undercut. 



SOURCE: 2000 mail survey of ARC grantees. 



■ Jobs created/businesses located in the region (16 percent). For example, a project with 
a goal of attracting new businesses and industry to the region reported, “From 1986 to 
1993 , our community experienced a growth of 12,000 jobs to our payrolls in a county 
of 8,000 total population, f,s 

■ Increase in job skills (13 percent). For example, a project designed to upgrade 
employees’ skills reported, " Unemployed workers were able to find better jobs after 
computer training . " 

Four (13 percent) of the 32 descriptions of project success were based on anecdotal accounts, 
while eight (25 percent) recounted a new or enhanced service that was being made available to a given 
population. Exhibit A-5 in Appendix A provides a sample of the narratives that projects provided in 
support of the economic outcomes that were achieved as planned. 

Ten (24 percent) of the 42 economic outcomes that projects delineated in their proposals 
were achieved less than planned. As shown in Exhibit 4-3, the primary barriers were the community’s 
poor economy, failure to fully implement the project, and overly ambitious project goals. 

Family/Individual Well-Being Outcomes. Of the 39 family/individual well-being 
outcomes anticipated by projects, 44 percent were achieved the same as planned — and 36 percent were 
achieved more than planned. Only 8 percent were achieved less than planned, suggesting that projects 
were generally successful in setting — and meeting — realistic family/individual well-being outcomes. 

The evidence provided for 1 7 (55 percent) of the 3 1 successfully achieved family/individual 
well-being outcomes was derived from a specific study or data source. The most prominent form of 
evidence was pre/post surveys or observations regarding skills, interests, or status — cited as the primary 
source of documentation for 26 percent of the 31 family/individual well-being outcomes that were 
successfully achieved. For example, one project with a goal of enabling adult learners to become more 
self-sufficient stated, “Theoretically do pre- and post-TABE test. Some learners left before we were able 
to post-test. Those who were tested did improve their reading skills. " The second most frequently cited 
evidence was a decrease in reliance on government assistance (13 percent). Six (19 percent) descriptions 
recounted a new or enhanced service that was being made available to a given population. Exhibit A-6 in 
Appendix A provides a sample of the narratives that projects provided in support of the family/individual 
well-being outcomes that were achieved as planned. 



5 Again, the inclusion of such statements in this report is meant to illustrate the types of evidence that projects provided This respondent did not 
provide any documentation that 12,000 new jobs had been added to the county’s payroll — or that any of these new jobs were a direct or indirect 
result of the ARC grant 
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Exhibit 4-3. Examples of factors that respondents indicated hindered 
their projects’ ability to achieve desired economic outcomes 



Anticipated outcome 


Factor that hindered level of achievement 


Increase employment opportunities for the most at-risk 
population in the region — the adult non-reader 


Unemployment remains high in Eastern Kentucky and 
business and industry were not as welcoming as we had hoped. 


Enable adult learners to become more employable 


The trend seems to be that basic literacy skills are not enough 
in this high technology workplace. 


Increase the stability and competitiveness of the manufacturing 
community in the 6 participating counties 


Unable to implement the project and measure outcomes. 


Reduce the likelihood of plant closures and layoffs in the 6 
participating counties 


Unable to implement the project and measure outcomes. 


Attract unemployed but employable persons to jobs that are 
open but which currently are unfilled due to a lack of 
developmentally appropriate child care for 0-3 year old 
children 


Due to serving less children than expected, we did not meet 
our goal. Considering we served 10 toddlers and helped 7 of 
those parents — that is a very good rating. 


Impact the economic needs of the five distressed counties 


Virtually nothing was done. The schools are closed after 
school hours. They would not open to permit use by 
community. During school hours, no community was 
permitted. So for all practical purposes, there was no 
community use of the system. It is a total failure in that regard. 
It is simply a matter of management. 


Attract business employers into Highland County 


Very rural community — no new companies came to the area. 



SOURCE: 2000 mail survey of ARC grantees. 



Only three (8 percent) of the 39 family/well-being outcomes that projects delineated in their 
proposals were achieved less than planned. One respondent indicated that their outcome, “Improve the 
quality of life for the citizens ofSDA One , " was “impossible to quantify. " A second project, designed to 
“increase parental involvement among targeted at-risk youth, ” reported: 

Perhaps our dreams were unrealistic. Many parents in this group are limited 
by unemployment, transportation, home duties, educational background, and 
self-confidence. We appreciate those who have participated and contributed, 
but the number is less than we hoped. 



4.3 Outputs Associated with ARC-Funded Activities 6 

Twelve (14 percent) of the 84 projects in the study sample used their ARC proposals to 
focus exclusively on the activities and outputs that would occur as a result of their ARC grants (see 
Exhibit 4-4 for examples of these outputs). On average, these 12 projects anticipated an average of three 
outputs — with 10 indicating that they successfully achieved all of their outputs (not shown in tables). 
Across the 37 outputs anticipated by these projects, 17 (46 percent) were achieved the same as planned 
and 18 (49 percent) were achieved more than planned. Only two (5 percent) outputs were achieved less 
than planned. 



4.4 Qualitative Findings on ARC-Funded Outcomes 

While the preceding analysis allows for a common measure of success across respondents, it 
fails to convey the underlying value of the achievements associated with the projects in the study sample. 
This section provides two additional perspectives on the accomplishments of ARC education projects — 
i.e., respondents’ own perceptions of their most notable outcomes (as provided on the mail survey) and a 
more indepth description of outcomes in four of the case study sites. 



6 Because Chapter 3 focuses on activities, we provide only minimal information about the outputs that were anticipated and achieved by these 12 
projects. 
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Exhibit 4-4. Examples of outputs that projects 
specified in their original proposal to ARC 



Improve opportunities for learners to find and keep a job in Whitfield County . 

Expand adult literacy and basic education services by at least 10 percent in Cherokee County. 

Increase the training opportunities available to literacy providers in West Virginia. 

Create a more ,( user friendly " environment for education, in the quad county area. 

Retain 70 percent of the enrolled participants in the Mississippi Mobile Adult Literacy program for a minimum of 50 
hours of instruction. 

Increase enrollment at the McCreary County Learning Center. 

Encourage leadership programs for Appalachia youth. 

Encourage postsecondary opportunities for the region 's high school graduates and adult population. 

At least 25 percent of the college ' s full-time faculty will be introduced to the use of microcomputer hardware and 
software for the development of instructional programs. 

Increase access to more computerized learning for students in Towns County. 

Increase students ' access to an enhanced curriculum. 

Build leadership, outreach, and decision-making capacity of communities. 

Increase communication and networking capacity of North Carolina 's Appalachian Counties. 

Strengthen the organizational capacity of Local Affiliate organizations. 



SOURCE: Document review of ARC education projects. 



4.4.1 



Respondents 9 Perceptions of the Most Important Outcome to Result from the ARC 
Grant 



The survey provided projects the opportunity to describe, in their own words, the single most 
important outcome that resulted from their ARC grant. As might be expected, projects varied in terms of 
the types of achievements they chose to emphasize. While some provided information about a community 
benefit that occurred as a result of their ARC grant, the majority focused on activities that were 
implemented or expanded. 7 Specifically, of the 79 projects that responded to this item: 

■ Forty-five (57 percent) illustrated how the ARC grant had enabled them to provide or 
expand a specific service or activity — e.g., serve significantly more students, provide 
a more comprehensive academic and vocational training program, provide a higher 
level of education to students in the school district. Exhibit 4-5 provides examples of 
the activities that projects identified as being their most important accomplishments. 

■ Twenty-nine (37 percent) described a general outcome (e.g., increase the 
employability of students, increase in the reading and math scores of the students) or a 
specific accomplishment (e.g., over 90 percent of the students finished high school) 
that occurred as a result of their efforts. Exhibit 4-6 provides examples of the 
outcomes that projects identified as being their most important accomplishments. 

■ Five (6 percent) described how the activities resulting from the ARC grant served to 
increase stakeholders’ understanding of the need for a specific service — e.g., 
increased awareness about the need for parents and community members to help 
schools, an awareness of the need for high-quality professional development for 
teachers. 



4.4.2 Site Visit Findings Regarding Project Accomplishments 

During the site visits, we had an opportunity to interview the staff responsible for 
administering and implementing these projects and, in some cases, the individuals who benefited (directly 
or indirectly) from the ARC grant. The process used to select the eight case study sites precludes us from 
using site visit findings to make generalizations about the range of accomplishments across the 84 
projects in the study sample. 8 We can, however, use information from the case studies to provide a more 



7 While the implementation or expansion of an activity represents an important achievement, successful implementation is not the same as the 
attainment of an outcome. 

H Case study sites were selected from those that were still in operation at the time of the study and had achieved at least one of their goals and, 
therefore, are not representative of the sample or universe. See Appendix E for a description of the site selection process. 
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Exhibit 4-5, Examples of respondents’ perceptions of the 
most important output/activity to result from the ARC grant 



Preschool Age 

Every county in the region improved the quality of child care, improved the materials and support for providers, and established 
a broad-based community board to ensure improved ongoing educational and health opportunities for children [ages] 0-5. 

Serving a younger population of children was most important to the people in our area. We provide transportation to and from 
home so we also were able to serve geographically isolated toddlers who were very much in need of stimulation. We now do have 
a building that can accommodate 20 toddlers and 30 pre-schoolers. 

Elementary/Secondary (All Students) 

To provide classroom space for students in the aftermath of fire (arson) destroying their high school (on September 2, 1996). The 
school system is grateful for your support. 

Enable the school to implement a program where students receive hands-on training in vocational-technical job skills that better 
prepare them for the market. The program is so successful that it has become a part of the curriculum for students in the 
vocational-technical path and they earn credit toward graduation. 

This lab allows students to explore technology applications to a degree/level not otherwise available. 

Astronomy outreach program (portable planetarium) use the van (bought with ARC money) to take the planetarium around to 
schools, summer camps, etc. 

Technology personnel and teachers in a seven county area received training in network maintenance/improvement that resulted 
in area schools having more efficient access to online resources and the Internet. 

Elementary/Secondary (Targeted Students) 

The project is self-sufficient and students are being enrolled for the 2000-01 school year. 

Serve significantly more students and provide a more comprehensive academic and vocational training program. 

Adults 

The most important instructional outcome was the incorporation of information technology into teaching and learning at 
Tompkins Cortland Community College. While such integration likely would have happened over time, the resources (hardware 
and software) and the institutional focus occasioned by this project significantly accelerated the process by which effective 
technology applications became infused into the curriculum at the college. It created an early environment for teaching and 
learning innovation that has subsequently kept Tompkins Cortland Community College at the forefront of community colleges 
within SUNY (State University of New York). 

Communitywide 

The ARC grant assisted BOCES to leverage a large project with NT1A (National Telecommunication and Information 
Administration, U.S. Department of Commerce) funding. 

Provided computer training for staff. Provided literacy skills for middle school children. 



NOTE: This table provides examples of the activities and outputs that projects provided in their response to survey item 10: What was the major 
or most important outcome (anticipated or not) to result from the ARC grant? 

SOURCE: 2000 mail survey of ARC grantees. 
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Exhibit 4-6. Examples of respondents’ perceptions of the 
most important outcome to result from the ARC grant 



Preschool Age 

Being able to serve more children was the most important outcome. Relocating to the center of the county allowed more parents 
an opportunity to use our program. When we were 12 miles out of town, we were not accessible to those parents who worked in 
town. Low income working parents used our program because we offered a sliding fee scale. 

Elementary/Secondary (All Students) 

Our students are more academically competitive with students in our N.E. Mississippi area. Students' have a stronger sense of 
pride in the school Community support for other projects has increased. Student* have a stronger sense of pride in the school. 
Community support for other projects has increased. Local funding for major renovations and additions has been more easily 
achieved. Other school systems have made significant efforts to modify their programs to match the innovations at Mantachie. 

The ARC grant attracted new and renewed interest in the museum. “Never before ” donors provided matching funds for the 
project. Two of those donors have remained financially supportive over the years. School children from throughout Alabama and 
neighboring Mississippi have benefited from the science and computer lab. 

The self-confidence of the participants increased much more than anticipated. Many students increased basic skills enough so 
they successfully completed the state exam to earn a high school diploma. 

Over 90 percent of the lads finished high school, went on to college, or got work in fields related to their teaching. 

Increase in parent/community involvement including self-improvement. This was demonstrated by increased contacts with school 
and increased participation in adult education programs. Improved attendance and punctuality on the part of students was also 
noted. 

Elementary/Secondary (Targeted Students) 

The increase in the reading and math scores of the students. 

The level of advances that students are making in this program, both academically and socially, have been significant. Children 
who once looked forward to the day they would be 16 so that they could drop out of school are now looking forward to high 
school — and possibly college. 

The families willing to accept help from outsiders to assist with their children. 

The students 9 responses to the job shadowing component has far exceeded (in many cases) what we expected. Students have 
consistently rated this component as helpful. We have seen remarkable attitude changes due to the community meetings at the job 
sites, increased self-esteem, positive goal setting, and an increase in academic achievement. 

Adults 

The number of folks who participated in the adult education program at all levels and the number who passed the GED test. 

Of the 5,500 adults served, 2 ,0 1 0 participants improved their ability to help children in school, 2,125 improved their basic skills, 
700 improved their parenting skills, and 1,100 increased their reading level by at least one grade level. 

Improved production employees 9 basic skills and for some acted as a catalyst for life-long learning. Many received their GED as 
a result. 

Improved self-esteem needed to achieve personal, educational, and career goals. 

Communitywide 

Increased student enrollment and positive job placement. 

NOTE: This table provides examples of the outcomes that projects provided in their response to survey item 10: What wa.v the major or most 
important outcome (anticipated or not) to result from the ARC grant? 

SOURCE: 2000 mail survey of ARC grantees. 
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detailed description of the types of benefits that can be associated with successful ARC education grants. 
This section uses examples from four of the eight case studies to provide a richer portrayal of these 
project-related outcomes. 

The Michelin Learning Centers are designed to improve the basic skills of employees 
within the Michelin Tire Corporation. As a result of these efforts, most of the employees who visit the 
learning centers have significantly improved their proficiency in reading, writing, basic math, and 
computers. At the time of the site visit, the purpose of the learning centers had been revised to reflect the 
reduced need for remediation across the three plants. (The number of workers lacking basic-skills was 
minimal — and new employees are required to have either a high school diploma or a general equivalency 
diploma and must take a 4-hour screening test.) As such, the current focus of the learning centers is on 
training employees in a wide range of job-related skills. 

The Adair County Technology Center was initiated in 1993 to address the problem of 
graduating secondary students inadequately prepared for the technical workforce of the 1990s. The 
primary aim was to expose students to a variety of technology-oriented career paths to make way for their 
transition into technical fields either in the workforce or in postsecondary educational settings. At the time 
of the site visit, the center was continuing to serve approximately 150 students per year. According to 
both students and staff, the center has increased students’ enthusiasm regarding careers in technology 
fields. In addition, 58 of the 99 students taking Tech 1 at the time of the site visit had already signed up to 
take Tech 11 during the 2000-01 school year. Course enrollment had increased significantly, another 
indication of the technology center’s success. Although no numbers were available, the center appeared to 
be achieving the aim of encouraging at least some students to choose careers in technical fields. For 
example, the instructor discussed three previous students who were preparing to study engineering at the 
University of Kentucky. One student went on to the Somerset Technical College to study industrial 
maintenance. He later joined the Army Corps of Engineers to become an engineer. Finally, the principal 
noted that the technology center was partly responsible for an increase in students’ interest in 
mathematics and science — as well as for an increase on mathematics and science scores on standardized 
tests. 



The School Outreach Project provides families of at-risk kindergarten and first grade 
students the individualized support necessary to maximize the likelihood that children arrive at school 
healthy and ready to learn. For the children and families served by School Outreach, the project has met 
its short-term goal of improving school performance, self-confidence, social skills, and interest in 
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learning. 9 The project director estimated that the program was serving 125 children at the time of the site 
visit — of which 75 were in their first year of intensive services. It was clear from talking with outreach 
workers, school principals, and teachers that children and families had benefited from their participation 
in the project. One principal noted that they might not ever really see the difference they make, but they 
have to believe that they will. Similarly, one respondent noted that the results of the intercessions may not 
show until the children are in high school, but that they must still work with primary grade students in 
order for the results to appear later. 

Another principal indicated that there were fewer discipline problems as a result of the 
project’s work with academic and social/emotional issues. She commented that many of the children 
served by the project are angry at their families, at their economic circumstances, at the fact that they did 
not get a Christmas present, because they do not have a warm enough jacket, or because they do not have 
someone to count on. The anger results in discipline problems at school. But by having one person who 
consistently spends one-on-one time with the child, the student is able to dispel some of the anger and 
work on academics at the same time. 

Principals, teachers, and outreach workers agreed that many students appeared more eager to 
come to school and learn than prior to their participation. Both of the principals interviewed during the 
site visit acknowledged that the outreach workers have been a key factor in increasing attendance. Aside 
from transporting students to school, the outreach workers have made students want to go to school (or let 
them know that they do not have a choice). Finally, school staff indicated that the project has resulted in 
better parent-teacher relations — e.g., parents seem more willing and less fearful of coming to teacher 
conferences. PTA meeting attendance was also up in at least one of the schools visited. 

Science and Math To Go! (SMTG!) is an elementary curriculum, materials, and 
professional development system. Staff and teachers reported that this approach has helped to close the 
gap between high and low achievers. Students are more engaged in the activities and enthusiastic about 
science. Teachers associated high interest with more learning. When a group of 20 first graders were 
asked how many wanted to be scientists, almost all raised their hands. According to local staff, SMTG! 
has allowed teachers to emphasize a subject that was typically put on the back burner in elementary 
schools. Moreover, science is being taught in a systematic, connected way, rather than through 
fragmented lessons based on teachers’ previous science background or available materials. 



9 The long-term goal of reducing dropouts cannot be examined until the children are in high school. The first cohort served under the ARC grant 
is just entering high school. 
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While there were no hard data on changes in student achievement, anecdotal evidence 
suggested that traditionally low achievers were performing better in science than before because the 
hands-on activities addressed the multiple learning styles of students. One teacher commented, “The kits 
ask students to think harder than they do for other subjects , and they like it! They leave the children 
wanting to learn more. ” Another said, “Lower achieving students shine when given the opportunity to 
talk and use manipulatives. ” 

Several teachers reported that classroom management is easier, even with the vast amount of 
materials and student-driven tasks involved. While more monitoring of individual students must be done 
with all the materials around the classroom, students are now more engaged and attentive to the tasks. 
This has resulted in fewer discipline problems in many classrooms. One sixth grade teacher in a 
departmentalized school indicated that students with behavior problems were well-behaved, active 
participants in her science class. Several teachers reported that students have asked to stay in through 
recess to complete their experiments or have asked to take items home to share their findings with their 
families. Another teacher stated that her students were borrowing nonfiction science-related books from 
the library for required free reading, where third graders generally select simple storybooks. 



4.5 Impact of the ARC Grant 

Survey respondents were asked to hypothesize what would have happened if their project 
had not received federal funding through ARC. Two-thirds (67 percent) indicated that their projects 
would never have been implemented without their ARC award (not shown in tables). 

The remaining respondents (32 percent) indicated that their projects probably would have 
been implemented using alternative funding sources. However, the majority of these 27 projects indicated 
that without ARC funding their projects would have offered dramatically fewer services (67 percent), 
would have reached significantly fewer people (56 percent), or would have suffered substantial delays in 
their implementation schedule (52 percent) (Figure 4-5). Overall, only three of the 27 respondents 
(12 percent) indicated that the scope, scale, and timetable of their projects would not have been adversely 
affected by a lack of ARC funding. Thus, without ARC support, even those projects that were 
implemented would have offered fewer services, reached fewer people, and been substantially delayed in 
implementation. 
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Figure 4-5. Percent of projects that believed the lack of ARC funding would have affected then- 
projects in various ways (n=83) 




■ Project would still offer 
full range of services 



£2 Range of services would 
be minimally reduced 



□ Range of services would 
be dramatically reduced 



Not reported 




■ Project would have 
reached an equivalent 
number of people 

^ Project would have 
reached slightly fewer 
people 

□ Project would have 
reached significantly fewer 
people 

E3Not reported 



■ Project would have been 
implemented on the 
same schedule 

S3 Implementation would 
have been slightly 
delayed 

□ Implementation would 
have been substantially 
delayed 

E3 Not reported 



NOTE: Percents may not sum to 100 due to rounding. 
SOURCE: 2000 mail survey of ARC grantees. 
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4.6 



Summary 



Projects were successful in achieving the outcomes they set forth in their original requests 
for ARC support. Almost three-quarters of survey respondents reported that their level of accomplishment 
met or exceeded their original expectations. Just under half of the 203 outcomes that projects identified in 
their ARC proposals were ultimately achieved the same as planned, while 30 percent were achieved more 
than planned and 13 percent were achieved less than planned. Skeptics might discount some of these 
claims of success as being based on self-report. However, documentation from the mail surveys suggests 
that many of these claims were, in fact, based on either a study (e.g., pre- and post-intervention 
assessment of participants’ basic skill levels) or external data source (e.g., students’ scores on tests, 
school dropout rates, unemployment rates). Further, the case studies uncovered convincing evidence that 
the efforts of the projects in the study sample resulted in a broad range of educational, economic, and 
social gains. 



Another important finding was that over four-fifths of the projects in the study sample were, 
in fact, able to describe at least one anticipated outcome or community benefit in their proposal to the 
ARC. These outcomes fit into three general categories — educational outcomes, economic outcomes, and 
family/individual outcomes — and cut across the different project types and characteristics. The outcomes 
outlined a skill or circumstance that would be enhanced as a result of their ARC grant. However, only a 
few of the outcomes described in the application materials had a measurable benchmark against which 
project success could be measured. Most projects used more general — and less quantifiable — language to 
describe their desired community goals. In addition, a small number of projects outlined a lofty goal that 
was beyond the possibility of a single project to achieve. Nonetheless, the fact that most projects had 
identified an outcome — and half appeared to have access to tangible evidence to back up their claims of 
success — suggests that a framework exists for ARC to build upon as it seeks to enhance its program 
monitoring capacities (discussed further in Chapter 6). 

Finally, survey respondents were asked to hypothesize what would have happened if their 
project had not received federal funding through ARC. Two-thirds indicated that their projects would 
never have been implemented without their ARC awards. The remaining 27 respondents indicated that 
their projects probably would have been implemented using alternative funding sources. However, the 
majority of these 27 projects indicated that without ARC funding their projects would have offered 
dramatically fewer services, would have reached significantly fewer people, and would have suffered 
substantial delays in their implementation schedules. 
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5. PROJECT SUSTAINABILITY AND EXPANSION 



A final measure of the success of an ARC project is its sustainability. In recent years, federal 
agencies have placed a premium on grant recipients’ abilities to maintain projects after the initial period 
of funding. Funders are especially anxious to support projects that will remain operational over time — and 
expand to provide supplementary services or reach additional beneficiaries. In fact, the ARC’s project 
guidelines stipulate that in approving a particular initiative, “the Commission must also consider whether 
the project will improve, on a continuing rather than a temporary basis, the opportunities for employment, 
the average level of income, or the economic and social development of the area served by the project.” 

This chapter examines the operating status of projects at the time of the evaluation, as well 
as the extent to which initiatives had expanded or generated spinoff activities. It also describes factors that 
influenced whether projects were able to sustain themselves beyond their ARC grants. 



5.1 Overall Project Status at the Time of the Mail Survey 

To determine the longevity of ARC projects, respondents were asked to indicate the 
operating status of their project at the time of the survey (Figure 5-1). Just under one-third (30 percent) of 
the projects were operating at full capacity — and 35 percent were serving a function that had expanded, 
grown, or changed. Less than one-fifth percent were in partial operation, reaching either fewer people 
with the same services (8 percent) or reaching the same number of people with a more limited range of 
services than they had intended (8 percent). Finally, 19 percent of respondents indicated that their projects 
were no longer operation at the time of the survey. 1 



1 As discussed in Chapter 1, projects lacking appropriate documentation and/or a knowledgeable survey respondent were excluded from the 
survey sample. It is therefore possible that the projects that sustained themselves were most likely to have the documentation and staff 
qualifications required for inclusion in the study sample. 
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Figure 5-1. Current status of ARC-related project (n=84) 




35% 



■ In full operation 

^ In operation serving a 
function that has 
changed/grown/ 
expanded considerably 

O In partial operation, 
providing full range of 
services but affecting 
fewer individuals 

® In partial operation, 
providing limited range 
of services to full scope 
of individuals 

E3 No longer in operation 



SOURCE: 2000 mail survey of ARC grantees. 



As might be expected, projects that received ARC grants more recently were also most likely 
to be fully operational or providing expanded/changed services. Specifically, 81 percent of projects that 
had received ARC funding since 1996 percent were in full operation or had grown/changed at the time of 
the mail survey (Figure 5-2). Of projects last funded 5 or more years ago (i.e., 1995 and earlier), 
53 percent were in full operation or had grown or changed. The effects of time hold true for projects in 
partial operation and projects no longer in operation. That is, only 8 percent of more recent grants were in 
partial operation, compared with 24 percent of older grants. Similarly, only 1 1 percent of recent grants 
were no longer in operation, whereas 22 percent of older grants were no longer in operation. 
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Figure 5-2. Percent of projects reporting current operating status, by last year funded (n=81) 



M In full or expanded operation 
■ In partial operation 
M No longer in operation 



Last funded 1990 
through 1995 





NOTE: Percents may not sum to 1 00 due to rounding. Three projects did not provide any information about the years of funding. 
SOURCE: 2000 mail survey of ARC grantees. 



5.1.1 Trends Regarding Projects in Full/Expanded Operation 

Approximately two-thirds (64 percent) of the projects in the study sample were in full 
operation or serving an expanded or changed function (Table 5-1). We found that project type was related 
to whether projects were still operational (or had expanded) at the time of the survey. Specifically, 
projects serving targeted elementary/secondary students (63 percent) were more likely than other projects 
to be in full operation. Additionally, projects serving preschool, all elementary/secondary students, or 
targeted elementary/secondary students were more likely than other projects to have expanded or changed 
since their ARC grant. However, the wide variety of approaches utilized by these projects (and the small 
sample size) negates our ability to unequivocally conclude that a given project type is more or less likely 
to sustain its operations beyond the ARC grant. 
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Table 5-1. Percent of projects reporting current operating status, by project characteristics 



Project characteristic 


In full 
operation 


Expanded 
or changed 


Offering full 
services to 
limited 
individuals 


Offering 
limited 
services to 
full scope of 
individuals 


No 

longer in 
operation 


All projects (n=84) 


30 


34 


8 


8 


19 


Project type 


Preschool age (n=6) 


33 


50 


0 


17 


0 


Elementary/secondary, all students (n=32) 


25 


44 


3 


9 


19 


Elementary/secondary, targeted students (n=8) 


63 


38 


0 


0 


0 


Adults (n=28) 


32 


21 


14 


4 


29 


Communitywide (n=10) 


10 


30 


20 


20 


20 


Project scope 


Single town or county (n=28) 


43 


25 


7 


7 


18 


Adjacent counties (n=31) 


19 


45 


10 


7 


19 


Nonadjacent counties (n=25) 


28 


32 


8 


12 


20 


Economic status 


At least one distressed county (n=26) 


42 


27 


8 


15 


8 


No distressed counties (n=42) 


26 


41 


7 


2 


24 


Statewide or multistate (n=16) 


19 


31 


13 


13 


25 


Metropolitan status 


Metropolitan only (n=8) 


25 


25 


13 


0 


38 


Nonmetropolitan only (n=40) 


38 


35 


8 


8 


13 


Both metro and nonmetro (n=20) 


25 


40 


5 


10 


20 


Statewide or multistate (n=16) 


19 


31 


13 


13 


25 


ARC grant size 


Less than $50,000 (n=18) 


28 


22 


17 


17 


17 


$50,001 -$100,000 (n=25) 


24 


36 


4 


8 


28 


$100,001 -$200,000 (n=27) 


30 


37 


7 


4 


22 


More than $200,000 (n=14) 


43 


43 


7 


7 


0 


Total project cost 


Less than $100,000 (n=21) 


24 


29 


14 


10 


24 


$100,001 -$200,000 (n=28) 


32 


36 


4 


11 


18 


$200,001 - $900,000 (n=27) 


26 


40 


7 


4 


22 


More than $900,000 (n=8) 


50 


25 


12 


12 


0 


Years of ARC funding 


1 year (n=22) 


27 


32 


14 


9 


18 


2 years (n=34) 


21 


41 


3 


15 


21 


3 or more years (n=28) 


43 


29 


11 


0 


18 



NOTE: Percents may not sum to 100 due to rounding. 
SOURCE: 2000 mail survey of ARC grantees. 
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5.1.2 



Trends Regarding Projects No Longer in Operation 



Just under one-fifth (19 percent) of the projects in the study sample were no longer in 
operation at the time of the mail survey (Table 5-1). Projects serving adults (29 percent) and 
communitywide projects (20 percent) appeared to be especially vulnerable to such closures. Two other 
trends are worth noting. First, projects serving no distressed counties (24 percent) were more likely than 
those serving at least one distressed county (8 percent) to have suspended operations. Second, none of the 
projects that received over $200,000 in ARC funding — or had a total project cost of over $900,000 — had 
suspended operations. This finding suggests that projects that leveraged additional funding were most 
likely to obtain the financial support required to remain operational beyond the ARC grant period. 

Again, it is important to note that we are unable to use these findings to draw definitive 
conclusions or recommend the types of projects that ARC should support. However, these findings do 
suggest some informal steps that ARC staff might take when working with adult and communitywide 
projects (as well as with projects that have not leveraged additional funding). For example, in reviewing 
proposals, project coordinators might focus on whether applicants describe how specific activities will be 
sustained beyond the ARC grant period. (As is discussed in Chapter 6, the revised application guidelines 
do not emphasize the need for projects to address long-term sustainability in their grant proposals.) In 
cases where this issue is not adequately addressed, the applicant might then be required to amend the 
proposal. Once funding has been approved, project coordinators might target these projects for site visits 
or more intensive technical assistance. (Findings on implementation barriers in Chapter 3 suggest that 
technical assistance might be provided in such areas as dealing with higher than expected demand, 
methods for maximizing the number of community members who do make use of project services, and 
making accurate estimates of the time and staff required to conduct specific tasks.) While ARC staff 
might routinely perform these procedures for all of their projects, these findings suggest criteria that 
might be used to identify initiatives that would ultimately benefit the most from a site visit or other form 
of technical support. 
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5.2 



Status of ARC-Funded Activities and Equipment at the Time of the Mail Survey 



In addition to assessing overall project-level sustainability (above), we also examined the 
extent to which individual ARC-funded activities within projects were still occurring at the time of the 
mail survey. Overall, 94 percent of projects indicated that at least some of their ARC-funded activities — 
and 69 percent indicated that all of their ARC-funded activities — were still operational at the time of the 
survey (Table 5-2). Only five projects (7 percent) indicated that none of their ARC-funded activities were 
in use. Once again, projects with a more narrow focus were most likely to report that more of their 
activities were still operational. For example, all of the projects serving targeted elementary/secondary 
students — and 79 percent of the projects serving a single town or county — reported that all of their ARC- 
funded activities were still in place. It is also worth noting that 83 percent of projects serving at least one 
distressed county reported that all of their ARC-funded activities were still operational. 

As would be expected, activities and equipment associated with more recently funded 
projects were more likely to be in use than were those associated with projects funded in the first half of 
the 1990s (Figure 5-3). Most (88 percent) projects that reported receiving ARC funding as late as 1996 
through 1 999 reported that all of their ARC-funded activities were operational the time of the survey. In 
fact, only three (9 percent) of these projects reported that only some activities were still operational, and 
only one (3 percent) reported that none of its activities were operational. In contrast, only 55 percent of 
projects that received their last ARC grant between 1990 and 1995 indicated that all of their activities 
were still active — while 36 percent reported that some activities were still operational, and 10 percent 
reported that none of their ARC-funded activities were still operational. 




5-6 



112 



Table 5-2. Percent of projects that reported that their ARC-funded activities/equipment were still 
operational to various extents at the time of the mail survey, by project characteristics 



Project characteristic 


| None 


Some 


All 


All projects (n=77) 


7 


25 


69 


Project type 


Preschool age (n=6) 


0 


33 


67 


Elementary/secondary, all students (n=31) 


7 


26 


67 


Elementary/secondary, targeted students (n=7) 


0 


0 


100 


Adults (n=24) 




29 


63 


Communitywide (n=9) 


11 


22 


67 


Project scope 


Single town or county (n=24) 


4 


17 


79 


Adjacent counties (n=30) 


7 


27 


67 


Nonadjacent counties (n=23) 


9 


30 


61 


Economic status 


At least one distressed county (n=24) 


8 


8 


83 


No distressed counties (n=37) 


5 


30 


65 


Statewide or multistate (n=16) 


6 


38 


56 


Metropolitan status 


Nonmetropolitan only (n=37) 


5 


16 


78 


Metropolitan only (n=6) 


17 


50 


33 


Both metro and nonmetro (n=18) 


6 


22 


72 


Statewide or multistate (n=16) 


6 


38 


56 


ARC grant size 


More than $50,000 (n=17) 


12 


24 


65 


$50,001-5100,000 (n=21) 


10 


29 


62 


$100,00 1-$200, 000 (n=25) 


4 


24 


72 


$200,000 (n=14) 


0 


21 


79 


Total project cost 


Less than $100,000 (n=18) 


11 


28 


61 


$100,00 1-$200, 000 (n=26) 


4 


31 


65 


$200,001 -$900,000 (n=25) 


8 


16 


76 


More than $900,000 (n=8) 


0 


25 


75 


Years of ARC funding 


1 year(n=21) 


5 


14 


81 


2 years (n=32) 


9 


28 


63 


3 or more years (n=24) 


4 


29 


67 



NOTE: ARC-funded activities were those respondents marked “some” or “all” of the activity was ever funded by ARC. Percents may not sum to 
100 due to rounding. 

SOURCE: 2000 mail survey of ARC grantees. 
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Figure 5-3. Percent of projects reporting current operating status of activities, by last year funded 



H Ail activities are still 
operational 

■ Some activities are still 
operational 

^ No activities are still 
operational 



Last funded 1990 
through 1995 
(n=42) 




Last funded 1996 
through 1999 
(n=34) 

0 20 40 60 80 100 

Percent of projects 




NOTE: Three projects did not provide any information about years of funding. Percents may not sum to 100 due to rounding. 
SOURCE: 2000 mail survey of ARC grantees. 



We also examined which types of ARC-funded activities were still active at the time of the 
survey. Most types of activities and equipment were still active — with at least 80 percent of projects that 
implemented a given approach reporting that is still in use (Table 5-3). In fact, all projects that 
implemented the following activities reported that they were still operational at the time of the survey: 
renovating existing or building new structures, developing distance education systems, installing science 
labs, providing family support, and providing career/college counseling. Only two activities that were 
specifically listed on the survey were still in operation at rates of less than 80 percent — i.e., providing 
emotional or psychological support (56 percent) and distributing mini-grants (57 percent). 
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Table 5-3. Percent of projects reporting that the following ARC-funded activities/equipment were 
still operational at the time of the mail survey 



Type of activity 



Percent of projects 



Physical plant 

Install/replace mechanical equipment (n=14) 86 

Renovate structures (n=6) 100 

Build new structures (n=6) 100 

Other physical plant activity (n=3) 67 

T elecommunications 

Install computers (n=43) 88 

Install/develop network (n=22) 96 

Develop distance education system (n=13) 100 

Other telecommunications activity (n=4) 75 

Educational resources 

Install science lab (n=8) 100 

Install other special use classroom (n=20) 85 

Develop computer-based educational materials (n=21) 85 

Develop paper-based educational materials (n=22) 91 

Develop teacher training program/materials (n=27) 89 

Provide teacher/tutor training (n=38) 82 

Other educational resources activity (n=14) 71 

Training 

Provide literacy training (n=3 1) 87 

Provide computer training (n=38) 82 

Provide GED preparation training (n=25) 84 

Provide job skills training (n=29) 86 

Provide parenting skills training (n=19) 95 

Provide academic skills training (n=39) 92 

Provide peer tutoring (n=14) 93 

Other training activity (n=5) 80 

Support services 

Provide emotional or psychological counseling (n=5) 56 

Provide family support (n=8) 100 

Provide career/college counseling (n=16) 100 

Other support services activity (n=3) 100 

Community outreach 

Provide outreach activities (n=26) 88 

Establish partnerships (n=26) 96 

Distribute mini-grants (n=7) 57 

Other community outreach activity (n=4) 75 



NOTE: ARC-funded activities were those respondents marked “some” or “all” of the activity was ever funded by ARC. 
SOURCE; 2000 mail survey of ARC grantees. 
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5.3 



Project Expansions and Spinoff Activities 



A total of 30 projects (36 percent) in the study sample indicated that they were serving more 
beneficiaries than was originally anticipated (Table 5-4). In most cases, respondents indicated that they 
were serving a greater number of individuals than planned, but that the type of beneficiary had not 
changed. For example, several projects that had relocated were providing training to more adults: 



Our center has relocated to a new facility. Our new facility has 8,100 square 
feet of space and seven classrooms. We have also opened a satellite classroom 
in a public housing unit. 

In the past 6 months, our program has opened an additional adult basic 
education learning center in a previously unserved community. 



Other projects were able to expand to include a broader group of people: 



More community participants now have access to a wider variety of college 
courses. 

This project led to the development of a technology committee. The committee 
has written and been awarded numerous grants to continue developing 
technology in our school system. 

We have expanded from serving 8-10 counties to 20-25 counties, plus a full 
time program has been started to take care of the needs of students on a daily 
basis in [one county] and the surrounding counties. 



Still other projects expanded to serve different groups or individuals outside of the 
Appalachian regions: 

English language classes are offered to foreign born and the center has moved 
to a larger and more accessible location. 

[It expanded to] all of our Michelin North America facilities. 

This year the USDA has asked us to create two similar projects for young 
people in the Mississippi Delta. Also, we have beefed up our year round 
program in the schools and in the communities. 
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Table 5-4. Percent of projects that reported expanding to serve additional persons in locations or 
organizations beyond those originally targeted, by project characteristics 



Project characteristic 



Percent 



All projects (n=83) 36 

Project type 

Preschool age (n= 6 ) 17 

Elementary /secondary , all students (n=32) 41 

Elementary/secondary, targeted students (n= 8 ) 25 

Adults (n=27) 33 

Communitywide (n= 10 ) 50 

Project scope 

Single town or county (n=28) 36 

Adjacent counties (n=30) 33 

Nonadjacent counties (n=25) 40 

Economic status 

At least one distressed county (n=26) 23 

No distressed counties (n=41) 44 

Statewide or multistate (n=l 6 ) 38 

Metropolitan status 

Metropolitan only (n= 8 ) 50 

Nonmetropolitan only (n=40) 33 

Both metro and nonmetro (n=19) 37 

Statewide or multi state (n=l 6 ) 38 

ARC grant size 

More than $50,000 (n=18) 28 

$50,001-$100,000 (n=25) 28 

$100,001-5200,000 (n=26) 39 

$200,000 (n=14) 57 

Total project cost 

Less than $100,000 (n=21) 29 

$ 100 , 001 -$ 200, 000 (n=28) 36 

$200,00 1-$900, 000 (n=26) 42 

More than $900,000 (n= 8 ) 38 

Years of ARC funding 

lyear(n= 22 ) 41 

2 years (n=34) 35 

3 or more years (n=27) 33 



SOURCE: 2000 mail survey of ARC grantees. 
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In addition, a total of 38 projects (45 percent) reported generating some spinoff activities 
(i.e., a new service or activity) since their grant period ended (Table 5-5). However, it should be noted 
that many of these activities actually constituted an expansion in the number of individuals served by the 
project, rather than additional types of services provided by the grant recipient or other agency. Examples 
of spinoff activities included: 

In addition , a college scholarship program has now been created for GED 
graduates to pursue college. Twenty $1,000 scholarships are awarded 
annually. 

Website development and web-based instruction. 

Team case management involving multiple agencies to assist a person in the 
steps necessary to achieve self-sufficiency. 

We are also partnering with the National Science Resources Center and the 
Association of Science Materials Centers in projects with regional and 
vocational impact. 



The likelihood of projects generating spinoff activities did not vary by most project 
characteristics. However, projects with the largest ARC grants (greater than $200,000) were more likely 
than projects with smaller ARC grants (less than $50,000 and those $50,001 to $100,000) to report 
spinoff activities. 



5.4 Factors That Facilitated Projects’ Growth and Expansion 

The 54 projects that indicated on the survey that they were either in full operation or that 
their projects were serving a function that had grown or changed from their original design were asked to 
indicate the factors that were responsible for their projects’ success. Fourteen respondents indicated that 
the demand and need for the services was still present or had grown — that is, their projects had expanded 
because there were simply more people who needed their services: 

Need of students for a broader base of math and science education to help 

attain job entry skills and/or enter postsecondary education. 

Increasing need of industry technology education and technical training. 
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Table 5-5. Percent of ARC projects reporting spinoff activities, by project characteristics 



Project characteristic 



Percent 



All projects (n=84) 45 

Project type 

Preschool age (n= 6 ) 33 

Elementary/secondary, all students (n=32) 4 1 

Elementary/secondary, targeted students (n= 8 ) 50 

Adults (n=28) 46 

Communitywide (n= 10) 60 

Project scope 

Single town or county (n=28) 39 

Adjacent counties (n=31) 52 

Nonadj acent counties (n=2 5) 44 

Economic status 

At least one distressed county (n=26) 27 

No distressed counties (n=42) 50 

Statewide or multistate (n= 16) 63 

Metropolitan status 

Metropolitan only (n= 8 ) 50 

Nonmetropolitan only (n=40) 43 

Both metro and nonmetro (n=20) 35 

Statewide or multistate (n=l 6 ) 63 

ARC grant size 

More than $50,000 (n-1 8) 22 

$50,001-$100,000 (n=25) 48 

$100,001-$200,000 (n=27) 56 

$ 200,000 (n=14) 50 

Total project cost 

Less than $100,000 (n=21) 38 

$ 100 , 001 -$ 200,000 (n=28) 50 

$200,00 1-$900, 000 (n=27) 44 

More than $900,000 (n= 8 ) 50 

Years of ARC funding 

lyear(n= 22 ) 41 

2 years (n=34) 50 

3 or more years (n=28) 43 



SOURCE: 2000 mail Survey of ARC grantees. 
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Eleven projects indicated that increased support from funders, including local, state, and 
federal funders, allowed their projects to expand. With additional funding, they were able to broaden their 
services or provide them to more recipients because they had the means to do so. 

The development of interactive classrooms in this ARC-fimded project 
leveraged the equipping of additional classrooms with funding from local and 
other sources. 

A number of projects also cited the cooperation, partnerships, and community support their 
projects garnered that enabled them to continue and grow: 

The community working together to identify needs and bet on them. 

The working together of all agencies involved. 

Other projects listed the provision of quality services or dedicated staff and leadership as 
facilitators of expansion: 

Excellent literacy services continue to be provided. 

The great need for this service , the willingness of outside funders to help with 
the costs, and dedicated staff and parents. 

Creating more accessible services or facilities and taking advantage of technological 
advances was also considered helpful in projects’ growth: 

Moving into town allowed us to service many more parents. 

The activities initiated by our project have expanded significantly along with 
the growing presence of the world wide web and the increased availability to 
new technologies for teaching and learning. 

With the exception of additional funding, most of the factors cited by projects did not 
directly provide material support for the expansions. These other factors suggest the impetus for growth 
but provide little insight into specific mechanisms that assisted projects once ARC support ended. The 
case studies conducted as part of the evaluation also revealed that for the most part, projects did not have 
specific plans for sustaining themselves. In six of the eight case study sites, the grant recipients took over 
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support and funding for the project at the end of the ARC grant, but their revenue streams did not change 
dramatically. (See Exhibit 5-1 for an example.) 



Exhibit 5-1. Adair County Technology Center 



The Adair County Technology Center was established to address the growing number of students 
inadequately prepared for, or even aware of, technology-oriented careers. The school-based center has 
continued in operation since the time of the ARC grant. This is due mostly to the adoption of the 
technology center as part of the high school’s curriculum by the Adair County Board of Education. At the 
time of the ARC grant, the Board was fully in support of the center and was aware that it would need to 
take over funding after the ARC grant ended. The Board provides $1,500 annually for center upkeep and 
repair. The center instructor’s salary is also paid by the Board. 



Several case study projects indicated that they were planning to seek grant funding from 
foundations, local businesses, corporations, or other public agencies. Moreover, two of the case sites were 
adapting their objectives to the changing needs of the community and beneficiaries. (See Exhibit 5-2 for 
an example.) For the most part, however, case study respondents were generally quite certain that their 
projects would continue. For many projects, ARC funding provided a dramatic increase from the 
shoestring budgets with which projects had been previously operating. They were grateful for the jump- 
start ARC provided — and confident their efforts would be sustained. Most simply planned to “make do.” 



Exhibit 5-2. Michelin Learning Centers 



Although the Michelin Learning Centers are still active, their purposes and methods have changed 
considerably since the period of the ARC grant. Now a part of Michelin’ s operating budget, with some 
instructional support provided by the county’s adult education program, the three learning centers 
continue to take in new learners. However, with currently little remaining need for remediation (due to 
changed hiring practices and remediation provided through the ARC grant), employees now come to the 
centers to use sophisticated educational resources for training in a wide range of job-related skills. The 
original ARC-funded learning centers provided the impetus for the recognition that the educational 
resources available in the centers could be used for many kinds of job training. The shift from remediation 
to training has resulted in a modified curriculum and a broad set of resources within the learning centers. 
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5.4 Factors That Facilitated Projects’ Growth and Expansion 



The survey respondents that had indicated that their projects were either in partial operation 
or were no longer in operation were provided with a list of potential factors that often contribute to a 
project’s end. These 30 projects were asked to indicate whether or not each factor may have contributed 
to their project’s not operating at full capacity. Funding was generally considered to be the biggest 
problem for these projects, with 43 percent indicating they did not have enough money for operations, and 
37 percent indicating they did not have enough money for maintenance (Figure 5-4). These findings 
highlight the importance of projects outlining plans for their future. In addition: 

■ One-third noted personnel changes (33 percent) and technical obsolescence 
(33 percent) and nearly one-quarter indicated mechanical obsolescence (23 percent) 
had affected their project. 

■ Lack of community need was not reported to be a problem for these projects. Only a 
small proportion (13 percent) indicated that they did not have enough participants to 
warrant continuation, while 10 percent cited a lack of community awareness and 
13 percent a lack of community support. However, many projects that were sustained 
identified community support and an ongoing need for their services as contributing 
factors. 

■ Over a quarter (27 percent) of the projects no longer operating at full capacity 
indicated that they had simply run their course or met their original intent. 

■ Finally, four projects (13 percent) cited other factors, including policy changes that 
shifted the time and commitment of participating teachers and a change in the 
governor and that state’s priorities. (Exhibit 5-3 provides another example of how a 
shift in policy can affect sustainability.) 



5.6 Summary 

Findings from both the survey and case studies suggest that many of the projects in the study 
sample were able to sustain themselves beyond their ARC grant. At the time the mail survey was 
administered, two-thirds of projects were still in full operation and/or had grown or expanded, and almost 
all projects reported that at least some of their ARC-funded activities or equipment were still in operation. 



Figure 5-4. Percent of ARC projects no longer operating at full capacity (n=30) reporting 
impediments to full operation 
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SOURCE: 2000 mail survey of ARC grantees. 



Exhibit 5-3. Science and Math To Go! 



Science and Math To Go! (SMTG!) is an elementary curriculum, materials, and professional development 
system that employs kits filled with materials for a series of mathematics and science lessons. The biggest 
question facing SMTG! ’s ultimate sustainability is whether the science and math kits will appear on the 
state textbook adoption list. If they do, state funding can be used for the purchase of new kits (and 
potentially their refurbishment). If not, districts will continue to feel the budget squeeze to maintain and 
expand their use. Staff report that districts are beginning to take ownership of the curriculum, particularly 
as it becomes teachers’ expectations that the kits are the districts’ established curriculum. To roll back 
now, when teachers are so pleased, would be politically difficult for the districts. 
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While projects appeared to have sustained themselves, it did not appear that many had 
expanded significantly beyond their original mission. A total of 30 projects were serving more 
beneficiaries than originally intended, although the characteristics of these beneficiaries had basically 
remained the same. Additionally, 38 projects reported generating spinoff activities, but most of these 
actually constituted an expansion in the number of persons served. 

Several factors appeared to be responsible for the high proportion of projects that remained 
operational — or expanded — over time. In most cases, the general demand and need for the services were 
still present or had grown — i.e., their projects had expanded because there were simply more people who 
needed their services. Many projects also reported that they had enough financial and community support 
to allow them to continue or expand. Quality services, dedicated staff, strong leadership, and 
technological advances were also considered facilitators of expansion. While most of these projects were 
sustained, in many cases this result seemed serendipitous and achieved without a high degree of planning. 
This finding suggests that project coordinators might use the application process to encourage prospective 
projects to think ahead about steps they will take to sustain their efforts beyond the ARC grant period. In 
addition, findings from the case studies suggest several steps that individual projects can do to increase 
the likelihood of success and sustainability. These are highlighted in Appendix E. 
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6. SUMMARY AND RECOMMENDATIONS 



6.1 Summary 

This report has provided considerable evidence that the Commission succeeded in bringing 
about a series of targeted educational gains throughout the Appalachian Region. 1 The 84 ARC-funded 
projects in the study sample enhanced learning opportunities — and increased educational outcomes — in 
the Region. Equally important, these projects contributed to the Commission’s goal of reaching those 
segments of Appalachia that are most in need, including persons in extreme poverty, persons who are 
geographically isolated, those who are unemployed or underemployed, youth at risk of dropping out of 
school, and persons with disabilities. The finding that two-thirds of projects would never have been 
implemented without ARC support — and that many of the remaining efforts would have been of limited 
scope — further suggests that ARC funding enabled these communities to address educational, economic, 
and social needs that might have otherwise gone unmet. 

The survey and case studies obtained considerable information about the general approaches 
and specific activities that projects used to achieve their educational, economic, and individual/family 
well-being outcomes. Because the ARC’S education portfolio encompasses a diverse range of strategies 
and activities, it is difficult to describe a “typical” ARC education project. The Commission enables and 
supports a myriad of education interventions — most notably teacher/tutor training, academic skills 
training, the installation of computers in classrooms, job skills training, and literacy training. This lack of 
a “one size fits all” model was viewed by many case study participants as an important feature of ARC’S 
approach to selecting and funding education projects. It clearly reflects the Commission’s emphasis on 
using its grant solicitation process to support the localized needs and capacities of individual 
communities. Through this process, ARC has provided community organizations the opportunity to 
address local disparities and take advantage of regional resources. Equally important, we found that the 
projects in the study sample successfully completed their intended activities, with most indicating that 
they met or exceeded all of their implementation objectives. 

Unfortunately, the small size of the study sample limits the extent to which we can 
generalize about the types of education activities that ARC supports. It also limits our ability to use 



1 Because of the nonrandom process by which study sites were selected, these findings only reflect the contributions of the 84 projects in the 
study sample — i.e., we did not attempt to extrapolate these findings to the entire portfolio of ARC-funded education initiatives. 





survey findings as a basis for offering conclusive policy guidance as to how ARC might further improve 
its education portfolio. For example, the survey found that projects in a single town or county were more 
likely to have achieved all of their implementation objectives and still be in full operation (at the time of 
the survey) than were those that spanned two or more counties. In spite of this general finding, we are not 
in a position to conclusively recommend that ARC focus its educational efforts on single-county projects. 
To arrive at such a conclusion, we would first need to systematically compare the implementation 
experiences (and outcomes) associated with small- and large-scale projects that attempted to conduct 
similar types of educational activities (e.g., by comparing the efforts of single and multi-county initiatives 
aimed at providing teacher/tutor training). While such an exercise might be useful, we do not believe it 
necessaiy, since most of the projects in the study sample appeared to have been able to implement their 
activities as planned. These findings do suggest, however, that ARC consider providing larger scale 
projects with additional (or more focused) monitoring and technical assistance. Smaller, single-county 
projects, which tend to be less ambitious, may not require any such support from ARC project 
coordinators. 



Finally, this study obtained information on the types of outcomes that projects anticipated 
for themselves and on the methodologies that were used to assess whether these outcomes were achieved. 
We found that four-fifths of the projects described a skill or circumstance that would be enhanced as a 
result of their ARC grant, although only a few of these outcomes contained a measurable benchmark 
against which project success could be measured. These anticipated outcomes generally fell into one of 
three categories, i.e., increasing educational attainment, increasing economic well-being, and increasing 
family or individual well-being. Half of the projects appeared able to use data to back up their claims of 
success. These findings, coupled with recent efforts by ARC to assure that projects include measurable 
goals in their grant applications, suggest that a framework exists for enhancing the Commission’s 
capacity to report on project-related outcomes. This framework is discussed in more detail throughout the 
remainder of this chapter. 

What follows, therefore, is a series of recommendations designed to enhance the extent to 
which the ARC is able to document its successes and enhance its capacity to provide technical assistance 
to its grant recipients. 
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